{"_id":"@microsoft/bf-dispatcher","_rev":"285-666e9943a019e194804a46c19ad34501","name":"@microsoft/bf-dispatcher","dist-tags":{"beta":"4.13.0-beta.20210316.e8ec340","rc":"4.15.0","latest":"4.15.0","next":"4.16.0-dev.20231122.4561d55"},"versions":{"0.0.0":{"name":"@microsoft/bf-dispatcher","version":"0.0.0","author":"","license":"MIT","_id":"@microsoft/bf-dispatcher@0.0.0","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"}],"dist":{"shasum":"21a7b1fda5b128908991e60245f4876ee440870e","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-0.0.0.tgz","fileCount":1,"integrity":"sha512-2tSzamop2N9f86w9GElHnnRxzFcH2ikjzYekcrL/PuXrPvwJ2FwObZCvCIFEW4nkNXIZoz5Bb02ntO0RDPDmOQ==","signatures":[{"sig":"MEUCIB//tiWf4PR/+pmXqK57cq3PPgIcie+cLrTflNmBsouMAiEA+8nTPskJf9V8K59DwRt/l3fUz8YARkF3X/Riqz5Qyb0=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":220,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfLeFTCRA9TVsSAnZWagAA3g0P/2DytsnyYefiy2uMoF0M\nVwTPdI7w1hokX93+VoN/54FBMoz3WZLCEoeeT4AuMENNN6Gtqr3+7zZPgzB2\n1oeVSsI65Bi4TzsVwwBtPLo6dGHaCw7MkM2JIGTYQpbWDfrUUSjPIuMfuws0\nBBrvrGcnKvoC+zaMZM4sICcA3JnpxMGdxORM1wqk3tAohFLSdQmb6KHwJVT9\nO5AFKgaBr91ia6sE/INWYdLFQVbQhv7jpfKMVqHbzcGEhSFItBykxb/AnJxq\nk0jY/giCrn+BZrgHG5I8mG0o0iyfi7np6g4ZSrRZhcN+PPyqqTxrfXoR62SQ\nJwD5AWP9qrjwfdrn5gYHnidt8u/A6p30CUsMhT17FCxmMzGTEyAbKA7s2iOg\n1b996N4l4jCAeFflG5krHfh4dQeqjOvI02XpbYMnW8CooPFOAUvszM+ksiX3\n0acFMXmx8L7NSwx73kyUaoiP2upsHtCRrBuwSJx0qFc+rEOlIXWMl2BWHWlc\nCVX8PoioBErYGzPNE00lqei97KeN190nZ6c/klmx4JHFx9QnWXsEvtUfCct8\ne4LquZf1lL+JDjh1e36mdvrOxYl7S13VignUwFC9ZJzk7SMpP0kB/gLnrpAi\n/8BlsVpHoiVcYSbi/OIgXnAoQ81YBK1kU9w7uMFatg62smbsBIeRYpQNMyYA\niBbj\r\n=VdY+\r\n-----END PGP SIGNATURE-----\r\n"},"main":"index.js","scripts":{"test":"echo \"Error: no test specified\" && exit 1"},"_npmUser":{"name":"microsoft","email":"npmjs@microsoft.com"},"_npmVersion":"6.14.7","description":"","directories":{},"_nodeVersion":"12.16.1","_hasShrinkwrap":false,"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_0.0.0_1596842322690_0.7863919240966515","host":"s3://npm-registry-packages"}},"4.10.0-dev.20200808.5a7c973":{"name":"@microsoft/bf-dispatcher","version":"4.10.0-dev.20200808.5a7c973","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.10.0-dev.20200808.5a7c973","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"4dc8b965f6080ccfc0196c889dbc47d828451a68","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.10.0-dev.20200808.5a7c973.tgz","fileCount":282,"integrity":"sha512-IAXpzNtlA4eH+FctdGsm5mQ60+Q2vJzj6cmCPElXGSAB3C1Rzif+WXz6VT6tk5xShO8grpmJcFXWP2/CpUfeqA==","signatures":[{"sig":"MEUCIQCVr336STURWJwCgrCyyNpr4IXRXETyZghUTu3haLf05AIgVnTiLOZdcT5OI16/TozzGGoB9sQkMQMEgz4nD0sJDhE=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1631401,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfLlHXCRA9TVsSAnZWagAAZ5cP/0Vl6C2f0v0YVDVfwFAM\nz9TQFCUxRDMTeW6Kl3yVoy0k3jzILKTtZbCELWHZ6EDCv6Eo5NFK+E5rAQcF\nLjVGwuWSS/LELprEn6jahizWiQDMwdBgpUBgek7AYOh5N6skCI6316F/cQGT\nVyz4VWS62Na17h0dTfuWTj45pkVkogIP2rW8yG/DcvWfwkFpOiz3Fe26Y7Nq\nhP/xoD2v5ho1ko4mkdwhBp9HDBKjp2YBS1u8qwb9zTBbnX2YoEqiSMmZmAuv\n/MVG4NQVSj9XUIf6fZVeIzhyWGXvr0lnYvSqumo3/N08vRsnFprhNLU/TF5e\nkR6PDY1QJOgn8saiAls3BfHGzEEP7FbY2DgB1IXSuRyMZHfGOaQBBBHz7Dxa\n36hp0m9E/kh396gLLQM3mltpVAgzMmvTOw0Bv0YjFnHLL7Ytne4xLNJKPNCi\nfvk7TvhtiJV2mE1kwPIk8l+/wbemeV4mIUOet/DwkIyjKYZFVd54KymygneQ\nbSGsY+qzls7cBOwuceVoZzv5QXvIQkdoZrLnTebioTE4cROKFNdyHaEdEJYS\nMy1FkCtdJLN5ZT80ztGaXfS6PWhf0Pv7VtzI6qBHQs75/Ko9nD+4q6FuFDqL\nzWqfuNVUtNJVHUvEdkLUfb1FUyQQ3014KwUcBs4DIRg2+yLZKQWMJMOY8OIJ\noQ8n\r\n=n47B\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.10.0-dev.20200808.5a7c973.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.6","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.18.3","dependencies":{"tslib":"^1.10.0","argparse":"~1.0.10","@oclif/config":"~1.13.3","@oclif/command":"~1.5.19","@microsoft/bf-lu":"4.10.0-dev.20200808.5a7c973"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","ts-md5":"^1.2.6","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","readline-sync":"^1.4.10","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.10.0-dev.20200808.5a7c973_1596871127087_0.660599292938155","host":"s3://npm-registry-packages"}},"4.11.0-beta.875403c":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.875403c","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.875403c","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"99cdc5099bdb01bafd94300247799adfd987ea0b","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.875403c.tgz","fileCount":327,"integrity":"sha512-de8JMBoqrfbws2UjziVCZNv+cqmmWD6CC8ltB7xHGj0a9GIc9cmlJbbm3eg8Y+fB2RuLbpkpcZtMRYz7689nGA==","signatures":[{"sig":"MEQCIFYvd3g+uTsmq5yQ3j0b6Rm6apg3GTEE6LC9cqMDkC7aAiAREuO4tt2jACpJeYJ2GSIIOVAzdPUY/NRq1tgCdzwRpw==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1782390,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfNGl9CRA9TVsSAnZWagAAbPwP/izvdEeb1ieChrah8K47\ncsZsRZ9P7SkY5ckpvXXR2Iud7Tp1oS6NJv0YeTh4KEv1k05/oBtGZtBe/PHC\nMBbV5eUBcD44Iqgzpo7ppZXAOmSFE0JqE2d1vYw4vP9i0oZU/cU1DyJgvNMI\nz/msz9MszYOMse664mysYG3KjsRixbQfTCCSq23htp/xDS76OBvr9h9CsbMB\nzwvrp8optcdOr6kn4M42Xv11tAzbqK+BGxU3GG4xz3Pp/MaZfV2okmz6YPfh\nNifEsyCaHof/GWkBfU6mlp/Z7Lmd0mwY/hkuYSJinuOaAci4F1vTBR09HA4a\nLjkyZUGbE4sQnqKl0kvL1Vuy07UVGZ61oK4ggURkyJMLSqgSxKzzdZgHTWxJ\nBTIx9dze52IyhmTthIupRZJZg0CICqqXUfqn9OqGiqTts09NC8zJ3mapvvNa\nUN5aw+4CkQMh100GnKGT8AyPfmNKDnyrlQz3rxTMUU3LKbuEFzdXhxWgpdZg\nzwRLMbjz9hCD1Igb3Iv611kw4G3Hp9QaRlE7SuV4vMauXNkqx5b5Z6Ucxlk2\n+OJ9A1e+71T+UQCSjhyFPCwv0/8txsSi1lQGnBY6Y240yx7vByJEKX8DGEuy\nOpmtF2NQsTXfOzRzOp4kXYHR2IF38H+UaZdt8COiHqow+gUFXFHwReAqucFK\naygv\r\n=Xj1m\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-dispatcher-4.11.0-beta.875403c.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.6","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.18.3","dependencies":{"tslib":"^1.10.0","argparse":"~1.0.10","@oclif/config":"~1.13.3","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","ts-md5":"^1.2.6","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","readline-sync":"^1.4.10","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.875403c_1597270396756_0.07046014303668868","host":"s3://npm-registry-packages"}},"4.11.0-beta.6e2953b":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.6e2953b","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.6e2953b","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"35d70a02f46794b723549691f5c884475cbfa19c","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.6e2953b.tgz","fileCount":327,"integrity":"sha512-kKftceqLPFsgB/9M9BA7iXEGo7CFrlNo57bgjO1zNHctLUfHw04VGzCkaOmkTo6X7yqQgWaUZm9cI3uCrnjmWQ==","signatures":[{"sig":"MEUCIB6KNluloXj5uJ4c/ir+GPLVPhkAQa8rY/SgZw3heFX7AiEAinAwgiVteSBjpe3aBicBNJH1u0U2B7bB+DNSlZ/s60k=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1782390,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfNIJVCRA9TVsSAnZWagAAv3UQAIg5o9zQf7DgKs9/Ay4H\nUHAWjRncnp6VRJpR27AHFJ+x5CFdupyBHumXFxA+8pmnzNjt3+V4Z6bvIBRt\nishx/R9dEYTwV+wdJvOxFF5UwWm3gJ6Mb9ZtEJ0CTHcWMfCZ1WzLTH+uNo99\nRi15u+q9CwQ4rWz0e8TKSKn4ZBDOuen0JPhqLfInWOJDixf0xiJsdRp7/oIN\nbci27ooR03I3ci6Ta1eLeyh6A8NQTmeNtccY+FgaspBXLkev5NUR5bnKwOtb\n5LVpxcDdr/HtrfHtg+99i5d4INZ+ogYT4l+m4Q/f/2clPVwho35fznPkTaEG\nLA1VNIu05dR3tVgRe1RchmeXcjahuHT2/c6Y1Cbc62jKDMBT+3w3XdDsVVw+\nO6hBgy12BtRn5dHjGEnutU5FHJUZvb3EjTaosCyyS4+DXaYqFoW8rXo7StId\nYR+L/I585WVp7scwzgfqPgE00fW2M/khiIOjqlSOsoa4OvSl1jcHwgndbQsL\nsCPBGmrT7sFh2UkUwZcOWjU0piiQGZgPDj4o7QyPFDpX4isqkq4uA8fFkOMU\n9V3nXKkdI5A+uTm22Nf96A538co+gsUJXiiZ49wJWW0h68iWjHeN+wCnydIS\nSy1YSyuyvmBWe0abAqAROEHnNWOQ24hTaMi7FhzRXHlUq7vJUZC9Jl+ifQ5L\nDUSx\r\n=bvOm\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-dispatcher-4.11.0-beta.6e2953b.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.6","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.18.3","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","readline-sync":"^1.4.10","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.6e2953b_1597276757234_0.9140794105938299","host":"s3://npm-registry-packages"}},"4.11.0-beta.6afce3c":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.6afce3c","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.6afce3c","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"4f7bf355f13b2ed8bf52e6778906b58d4beada44","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.6afce3c.tgz","fileCount":333,"integrity":"sha512-yQ/9aNLtKOerSed2+1ZigEXNbFKb960kMKh5ZC/I6+wU+vQQRFabG6gd1NOXcCvqAamysrZ4kckowC1/tsmTWw==","signatures":[{"sig":"MEYCIQC7p42PNL8E+Hk4m1sgOVzsO06EKSePz3p0TaqEyw5NoQIhAO3n2Tr6hIorPEj/YKRK6OtzDsPCNN2kTQOxxAEoiOd2","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1795487,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfNYtiCRA9TVsSAnZWagAADnUQAJqL6dq4YEFvpU7KgzZS\nG0e5XXOMsVGrlM/bvkqf++I3O+eiyGR4w2ixdiyBAtmpk0HvrSDt32W2tbGX\n+qOqo1BaSbPYCUVmH9WcP0xmx3wYfSW4EHfmqiv3VXeRl2IJpdtoky2NTQF1\nmO1xDzD036KIm6YY2rtduyaYpH/NKS/b5tPduUV9ZUU22ZDSY4sk4R+HOEga\nzwW2Dos9EGcgD9Uyo1cHoHKE5xMzwP1515cQcaOdPVMh3YuGHd4HuDthCSqH\n9KbCsFmJdqKWUgEx1P1fiusyGflJhW5qlAkBaopqPgTLKDaY+adAeWLGZSzq\nNh0Lgh0wSUKH/vwI2rVhbva5u9yIJHBsjF+A2dPLBs4Keh+H2JnDekT3bbIL\n8+me9xAK1mgygFgeBBIK/ovDXN0atbzftu88gpIJg/UguNtWhfBoFRREiF45\nSyqNBmHwu07S6NE/OpvVBPKVZL55GnMZzo+7uSvKElGwXU2izL1ku2soLwis\neDOhbirs/ahM6Q03hWV0SuXrkvG8/Rp8SOAGih4o1GfSpGq5yDK+x23lRZD4\n9VKVc5CSctbegGFbROQv8sLmJu1Tc4X7Jc4hkqYdt0kdC1HvMkCys5bdQdXx\nnpejSSiYvPzMHs9OpTKZVWCMQiDiKzl6R+SujddnSlwqr8xQ9wD+4j8WoObP\ndgvP\r\n=hNTf\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-dispatcher-4.11.0-beta.6afce3c.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.6","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.18.3","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.6afce3c_1597344610207_0.9113223032837372","host":"s3://npm-registry-packages"}},"4.11.0-beta.20200818.8213b2a":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20200818.8213b2a","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20200818.8213b2a","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"43f780e26bbe8dd70e52e4fbb33438280d5b6f38","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20200818.8213b2a.tgz","fileCount":333,"integrity":"sha512-+4IIsqIrP0mCRfMVwOJYQy9A3sAt3MkXbRKxhFp45aMeWnIbdOrFRaq+Qx6Ol46615FoEK6VTq06rcDmZNA5zg==","signatures":[{"sig":"MEUCIQD+majDWymxCCaTQMFDxqRlVzwybDnO126bdgINDgegdwIgVh3/ucnSiODFhNNuY3JU+CXHXViTHQuSXgu8Agk3beg=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1834752,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfPDgQCRA9TVsSAnZWagAAclgP/0WSz74yyOH6F/ymQ78E\nfi++SpwfI63NePIvhsWkOZtYeoaziavne2XgqM4VVs96maEqk6MRmYxwpW4h\nJMgGUedEpYVzWqeKjOw2lk+oWvEE5RIpCvznnMTiO3yN2lypNT3FuDSfnxEC\np1imPQ/7YHXaBxKsU0CLmJxyBxkoI1jzA5v/aiRZc/Q2rjf+Jy1BtO4MHzg6\nIxsANveghML0dRNpf5/z5kyMql9c6kTBNqkiGQi2UIBInw7VjfyWFTD5TLVr\n4V1pG5fctsNBnTZPDZQRvuGzEA5J4SAy4ygsYMt099nfHIKieG7bpw5UIVxH\nn1U5pLic59cT7HWIy4bOrkidyZ+fBJ5DsU+8y93g1ged0sc6ZYUsM8d7E7CK\nP3EYTffWUrL8glTKI+PW4qi3S6IYwfq+9Hl4fzEtMyvwqr92tsB99TR5ebf6\n0yRCNQKhZYfHQQyaV0+JOfV/L5VZIfF/H2mzNbfP++S7mxExvqlJGmljdRoL\niy3KJ4yvseZNblBQCGPmrscJzElA0nOAuBPjWVhkTJh7ZTbj9uIvtJH04/b1\naZiuPUINwncxC4qKY7giZejyEPeCiAiYfzX7QdGSWkeNZorBQYhGICx/TB8R\nxJXFM5Nlx6gWmADAYwB0Uawc4izQeIDSvQ/7J+10BjbOra1djzMMowIx0eU1\nIwpS\r\n=HVWd\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20200818.8213b2a.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.6","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.18.3","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20200818.8213b2a_1597782032269_0.03906504606954875","host":"s3://npm-registry-packages"}},"4.11.0-beta.20200821.21f8e76":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20200821.21f8e76","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20200821.21f8e76","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"290bf33ef67446020fb20368ac559c5f96e6b9f8","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20200821.21f8e76.tgz","fileCount":345,"integrity":"sha512-cz7foKHOCVzxDRruVGfF5LDLxBO0GX+U4AmeAzGcqAOE7eB/Ynzy0FKw1cjekzWx58GeJDWdDdeKz9TdyyWzsA==","signatures":[{"sig":"MEUCIErKlxEm25UbFA8asBvIIIIAriImlprMnhd70oOrF+U2AiEAztz5hdh/JcYJLrajvP8+kaxHgPnh5KjV07Mnfyw97GQ=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1854878,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfQBbFCRA9TVsSAnZWagAAskoP/jJbWx0XKrI+kb6QKIQv\nX3gxHnpC0SYpUMRCRttrnH10R4W3HbIUK8DvoPnxqJ+pbM0lgJS9+dSeihLs\nRpg0wxshTFkK9JzuMxOfMg7V2/Bk/mPuY7ccXpww2yrV0noy8de/8wWEDcZA\nttipxZzPV/Htbs0FXsSFE71FKTf2krIAUBpkI74OsRmtBEy98k3NxqMpr54M\n6r7FWQThRWDGrJWliZfMli18ayFrUZSrv8+a1Y0ufckSvh/6sm8EiWRV3Hd1\nbTwUnishk9KgKRzKQv9AsrJKtJxTf1NTp0IbBTWeIXsanu9YC52VJDHmP0R2\nlu6RU2uFS+0RBE90HMbz+sAZD80d1vvyojH2nWhRlSCr2Ahv6wBn9kFiFqL1\nf/02/jSJjY7mUJKdYIsUVad9WH+8dwklOjmClfW+mhucDHxg7buFHmXIx4LT\nZY5frvwRag0xsn4Tcu4/E3H9DFPygE8lta8jyODzhMtQkO4kRPMQla67qiIb\n9DZqW3aX7Fy2ABZheOGrtmFAFcOcG6X8xGcmlGGEpuot8R67GjULUfzFsHAU\njO9fZNpawRLXET/CkvrjiLhnfO5Rh7i3U1J40bCmp9AVZhn4ei6wwCEvbea9\nWUAjR/VKTQJCGZ9L1QXlHjRX+WglvqFuREpyt3RnfBft7tECBLnQ3oJUA9gi\nnEx9\r\n=/vLO\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20200821.21f8e76.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.6","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.18.3","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20200821.21f8e76_1598035653073_0.012606729021902074","host":"s3://npm-registry-packages"}},"4.11.0-beta.20200824.6c63bf3":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20200824.6c63bf3","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20200824.6c63bf3","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"5a6ece6049531338d6c1b0116576df7fc03918bd","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20200824.6c63bf3.tgz","fileCount":345,"integrity":"sha512-SLIlUk5Qqi18J2j1cLFhEiVDGntgQAG9Io+1yAB1tAXL6hQnBAp5VJ8vAtPSAmLDNp82M02Mzl0EpRdcac6EiQ==","signatures":[{"sig":"MEUCIBMJi2GLqxWYNEwullknRq8xm4k+F2G5slk91aMzo4B2AiEAgDWPFBDLx6TP9rcZvrj78P1IBT1OW5ApRGcYhZ8bdsQ=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1854878,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfQ2d3CRA9TVsSAnZWagAAg64P/0tuUoxWaj4q6qlbIqwv\nphQkxq0LFUimGEAPS4Nh+PL8zCFEHQ2TRu6A0ZJyLmanUFmy7+JuZFplVUCT\nHgaybhTY0IW9Tu0Pk+H0eQla1IKbNtz4U3UCE2b54C/uIfKVryPCjOEFVf4C\nphHVQebvoLFJZ8Q24gWJv2egIefd7MbrAu3Rs0GGSI++WeC3okZCjUcRvaCo\n+ykNrHcXbnuvvyIt1uVoIAYBqa7dnQA8aXxSuQxtqx8tFECgUE82PZEf9r1M\n/2vcbR5NQr+iGkDNB/v3vFk075Y9YAHqFhF5s1h+Lq1j3dwRMmR03Id53nLb\nUHjchkiLTC7bN2ndyO79CfkWe/0iImZb82VJHUGz4W/jNJlyuzMJOWcgfCV9\n4ma7bn/uB8Bat8o9uE55DCI7g0FlmcE6w2/hxVSC/5ZrxH7FFsB/NoOjc/MT\ntZrdmFvHVDEi0qB8gnaH5bFRjBimC5lCjUnU0jEbMJLB0Fzjti9upHvTj5tl\nhdHBkSjsiTYrET793A8Y8lMch8OQMSXxipm5CgYxQj/r24F1RK5CDF0LGukr\nSFs0XfVoFmtpg+hkMvjHP8Lj3Y2zs1fzbZ4fpDIKmuSjUAQlBGyj46xraiaw\nvS9n/nFW7wWWQr8UWvs1oAa9j/XKOLAkAlZtPyHV8JuxoqzTaKf6w9eTKB9O\ncl+/\r\n=AH+j\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20200824.6c63bf3.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.6","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.18.3","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20200824.6c63bf3_1598252919302_0.0033476523709257133","host":"s3://npm-registry-packages"}},"4.11.0-beta.20200904.e677869":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20200904.e677869","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20200904.e677869","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"85f3405ee4862912a02add9215e07ee3ce3e11b6","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20200904.e677869.tgz","fileCount":360,"integrity":"sha512-CHKjQDYCKRJurX+wu2W7AaEqw3JZQzIr4Lq4wOTFDjTtSoBJ4+ZVVsbPNpF4fgAGCDq/bMNg/uBiDUidp5aI7Q==","signatures":[{"sig":"MEUCIA8KIVB0IA6SMN464qRg/zlujPDQwZM3nmQMc0apK7leAiEAscOef/Qh9xzteGaeQzAVrJrmpSjl+WcUrAGACiYpgaQ=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1914316,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfUegOCRA9TVsSAnZWagAAyToQAIVhZowLppPAixD4Fg6B\nn5VZ/y9axVw8YGeRKBrspfoYtpt1MeBAXh9lyAA4iSinRsrhHh76iE17L4vG\n1ScC/Th+wfU5yVcip/t1oczVnG7zI1gsCoRX60UjmGnVINL7jF6asmkN3IsL\nEyDRdi4whHUHpUwzpfZHc6lsa02gLl0jR8cuUyCt+HPijR0Z83Wq+2/kDjtW\nf5rYXIRfJ690anCS1G1/lnEX65mRwsgrbCKPDBijeSKfnzba6LjptihfT2+6\nmb5b0JxIHYqRx0fkezVa2zAx+2tWUROME4VQjJhfRhw9TKOeRL/hvZqcH6aq\nf18bhpBt9y8V8fV4lMdTmtrxlAKTOUW5YxYOcw7/aZ3LK5YDr0U9AX6Y2Idq\nv1invdezuAZngl6owr5OwlA2vV4dMm5l1XCIQEUkDLsoXjJ5MVf/Hjdnz9y4\n00tjHPyEszrtsk2mIHcSMzeNEdgtmcmRc/5Judcej90w2rmdfJkdWLfnluGO\neSJu/AlifB4Dui0+E1Bk/NTfvvot7POchO3LscwLej/0Zyy0RLv7p1Q4bvYu\nlH6BmpYsOAjbgo5Z6dkreWdhmDk4QMY3qXeScN03idu0WxvVZUfcxLPzXh3d\nbxxorxKRfpAWWq2UfMovU3G5Czptn7ZoYL/uR8wJdpG10y89gdnflnBpbKwL\nHfId\r\n=ugQ+\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20200904.e677869.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.6","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.18.3","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20200904.e677869_1599203341850_0.8308483759049683","host":"s3://npm-registry-packages"}},"4.11.0-beta.20200918.ad209cb":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20200918.ad209cb","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20200918.ad209cb","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"520695dd8fd04270e7008efda070f3f480afe4fa","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20200918.ad209cb.tgz","fileCount":360,"integrity":"sha512-PbhqLDy+DxsaFU3+/TWyML8hyg4blTs2jLBI1pi538yMIxsj9AHF4+dZ9yHH+qg0dW4qt2Q5iPJHQqvocNHnNA==","signatures":[{"sig":"MEQCIBbJC0g0arcuhqjEj9COWDDhIQGC4adPmjqXhJy1QaiIAiBvd5GiBwQkb0gKgfhOSkzBgAgC+J9yxpxmV4xDezDx7w==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1914320,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfZF0kCRA9TVsSAnZWagAAv94QAKTcgKF00ywDnShkC4+i\nc6gFRLOtxWDcUwvv1SiZ4y4iGbiY2Fs1fjaXMAZzy9nEbmuDdlOCAafOB1Rq\n0FPrzoyT8QJfBJjCNz5YbJH8U2mgnTLvoHR/5GmN0llOl0yoMDuoq0Q0Zmxp\ntXsjaDXRNV9DGRayFJD/Usa45Cw/ULwZbCkqDQxDRHHkcqYlS0t3w8vexURP\nV3BkXJCnAU65qKQZSabQ1C4ksbCd3V44VX1uYdG8ycwYQs9aLY8ZfUQcMEo2\njbW8nNJSs6el2h3FAuD53Tu6soa8rRYD6aJNsl21dToahCcD+waJ9Aq1yf/e\nNkcp3N/fsVxCi0vfibC3mm2y/ZiAgq2zmW465uFCt1Ww2pNY2rMT16Pixl3X\nd+x8B6OR9XmZK0WWLfLcdpoFcI4BFLikgVmnkW+fm7gyNL+CA4RLaAtc9emi\nBMyOhnMEd+6FiT5rpqEFIbtIXZDxdu2GCDk8yEZ53KraMQGrcCb/aW2zbaIp\nrEK1Dhui1NvIG9jy8KmgPtHXk496H09vXIHpWYK1+rpSm3zcdMV3VPWlgdLG\nEY2RRytWNUvfQl/p02xclq1BWgUWFB/dR0cLyPFY74M4A36U10+GUPmPLK/e\niOcsd4P9OWek/M6j39uv4iH9Iuzgxn7jXt5jukaFvXwpCNelJzgpf6onU4F0\nYV4N\r\n=YcnB\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20200918.ad209cb.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.6","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.18.3","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20200918.ad209cb_1600412963424_0.05370516297963923","host":"s3://npm-registry-packages"}},"4.11.0-beta.20200919.b7fef29":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20200919.b7fef29","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20200919.b7fef29","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"5bb888b8f07a50266c0c9a950553cfe489551d4c","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20200919.b7fef29.tgz","fileCount":360,"integrity":"sha512-2sxUbZSkDx15BQj+JH34IMS9A+KDpg7tD5CyU4sh4XMU2ET1SyRwekbgGjbHq2yB/w8+vXGeoHqpEyagZERPFQ==","signatures":[{"sig":"MEQCIHgFkGbzFD7vfjsp+LlxN9fxZGfRDJpW3Q6ta/RmYeDPAiABUktaP2FV9+DV7TEJP/2bBdovk4C1k1i/I2d1xbCPvA==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1915653,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfZa5gCRA9TVsSAnZWagAAZEQQAJZsi+Z0USQZJdv17FPw\na8ZfoZ3ihLrtm1HFw1nHigfjx6UILPRLUYNht+Dp9E4gBMqAbdVyR2sNnZki\nCouCPkwE9cnUA+4p+M3iquWFup4TAApyyTGLhfLEuwKY5dxMOS6SuJtrxG+z\nOiH2XZMHTvrHtKyRUCl0Mhy9b4svaAcnCWpP/DPStVDIvwipggYspjLmtYm1\nQv8i2Wtwleswishm0cbhiesecoy5+xPxo7t/Km/A5Yif//6HCDswoovRl6nm\nEXtnCCT//BBHroBvYycoWT1wofkWjUFxQwqtmNwYqL0zoYjIgVzM631BgexE\n1LFOx4Hu5FOipjm1ntOfh/etFcpcwlbOzDynV2kOw4w3q9QjdAJDN4yN+GD6\nnCm+VmBWg8h4q8r4U2U9fwaMcyyU6LQ2nJ3z55+g2rC27Yq5m/UQORacO8s1\nz5KMrB26/sqdXS/1vzIoEkCAoqjcC1YQjOSjmg4J5p/etVWe28c5ec3PvG6A\nNTZ2qwsidrXaPRH/KvafTUwFKcEqQO3MUWnVJThHWXseZ8KFid2+WHG4mlth\nzrRxL2PfUcJx/bvU8K0rHFx0dxU59m4MLFe0vRd9dHlMzTUyPuFnin/jeSu/\ngv8tXF/e0rDMxNrikeWHRS294kYAByWfY73Js+d+AWIlU3OHM+0t6GjGwgYX\n+DmA\r\n=7VVE\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20200919.b7fef29.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.6","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.18.3","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20200919.b7fef29_1600499295718_0.6580824130444851","host":"s3://npm-registry-packages"}},"4.11.0-beta.20200921.2320a67":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20200921.2320a67","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20200921.2320a67","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"d29a4cb39c00ad268292b0630f8d0110823a175d","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20200921.2320a67.tgz","fileCount":360,"integrity":"sha512-S/E9EwCwn26CLssDrg3JZJPZbfIR2daFAnaSlfriTMSXPoESOSS/ZUdKtw9kETKj2fmmHiv+cjnE4772wdLVwA==","signatures":[{"sig":"MEYCIQC5PCBF4igLTNPSr9u4EVFL+I+J5qyGMbbMxC4o1OKLrAIhAL9NivD7HxR+XhRg5akFEtMLN0M9xKFg96lwkpgEr8Y2","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1920267,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfaFFzCRA9TVsSAnZWagAA2BUP/is6PiG+mx3h3HOT1Dzp\ndOV3RvuIj27Jx/UVYpGjil0r5v3uArCCZ7aXm3Ohd9aT2kCCjGBsTCi0M/ke\n92cD59OdGRVs2KuaXmMIpCqaSzZ+3IJuru2kBfw+zVken9noPuqhp2fRLu+7\nc7Cj4e0K9falL2NVH4+BbNLkwr2x0FSZfTXx58MgZ7c6ldoLRWqBz3jUEyxf\nFmn+d66wO8rda0sRQvMBf1f/B1dcLnGaxsqG3xG679MbPwjp7KB/j+3iNsJE\ndbJ/1mQTbOHIesvShxbN114yI2V5ZtOZ4CrERmkOJVl31UcHkPso37r1KVTJ\nBKcQChjQI36SOfSDgFJbPtwtPmTdvrWojkX9O88dAYtd7hyotpuE/qi6Y7gB\nOteUuF645C2khRfgBn4qcvPFbQ6U6sbOCRVVtsoJBNesyroC67vNjK2nA8Cs\nQVqETpnHZBfntXlSPOv5GiZFfqyrPxXxdMv68gSKzlY4hFwYrXMYuvfcVg3I\nmo/Cybrr4NMEchTC0745nIhXxDYKCDbVIzpUCzFwit7Zj7VTq0nEQ1bZK8Eq\n4JhL7+FpSkM2K0GavYGja4rB1DLnMTC8NMpc38UFGYY7btG1R5DI8I3vgnMf\nfYU8MpU/L/lEFkGVELuRv6YAkyTnYs21y5GunS3bJ+SUXh6/N2Csp+wpdqzW\nfqQs\r\n=WJDT\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20200921.2320a67.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.6","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.18.3","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20200921.2320a67_1600672115193_0.9609248651850724","host":"s3://npm-registry-packages"}},"4.11.0-beta.20200922.a071f5c":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20200922.a071f5c","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20200922.a071f5c","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"110923f6448d6a4e595a97856e856c6fbd4ad1f8","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20200922.a071f5c.tgz","fileCount":360,"integrity":"sha512-lvMOtXW4kzkbcrl9NNlJYtefv+o3dE3y283nI1c+oK3E4VDLYWYtzu/3uFqsFWF5125vf7XO1YjZcbUjOpjrwA==","signatures":[{"sig":"MEUCIBEv3P2oFjLyQ6kTCuGRNzta8D/tc6zBwnSmUGltqOehAiEAvD75AaFZJQpeJfGMB1oTHWOowOM0nRShNA6EaFAn8OQ=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1920267,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfaj4bCRA9TVsSAnZWagAA4sQQAJ5dVbEhWCKtQ4+cq2M9\nswy2lDMWdgGLaxgA3F+4P8e8oYwXxmUMG5UMTU53UV4DI9O/fMl64mYDI5so\nhZII1SPagThkCZy1U+wK5gdFdQ9yt1ZCnb/AK6mL5zDj+YI5DmVd+3IIEoMs\npXI7PCbRKvTidnuE80/oyRK+4VKNtX1qoInw/k+HgBD/9AiDn/UYqiu5QUyv\nTd1zvWfFZ6ZSG5vLnB50xR7lE0jKvFtS6RfbB3qrgyb+ZQQNIx9yMWkIANjE\ncbWhwdfFG8CkNeRH5QUA49xva2owE33FgjV2nzU/HlZXM4ZLiPMwELyiCw8o\nTwhHsdqI7iHkLANBvfKuuMUbHYinFme97v6YltzOz/aKiHk3q0wlSznM3vs0\n5ymtRFXWFJsUay4E9hwhLmsrGh8wNBhfghvJ4cAu/5hrjUFjvkQ194cLrobd\nAVoEuhd/pCkDROIKoCn3sFg+iMiKm9drI6ib3DkOmI2KUaFge20Dfe3fHtKu\nr7+x7PocSlNFC893OvTCVJIge/gBD3YU4Xxe7jg1QpFScIuAczr7R1GWxSYq\nDQSrKXH9V8ZaZSWEhUABBIq4l5369+sxfuSOH3qSpGTcS+qUT0RZ4vdl4yUX\nKxA2vCXXfEtyizKFORdL+fD23/79fHSeOCT6HcNTxcqiWc3bnmuswxdx0Vmg\nolpg\r\n=4DdU\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20200922.a071f5c.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.6","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.18.3","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20200922.a071f5c_1600798234853_0.45465267206209825","host":"s3://npm-registry-packages"}},"4.11.0-beta.20200923.7e94eb2":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20200923.7e94eb2","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20200923.7e94eb2","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"b737951ce82bf03b9ed3b89c619799a50697226a","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20200923.7e94eb2.tgz","fileCount":360,"integrity":"sha512-s/0u/beNV3ymvfe+OtMYrfmRumgXwS2uHzMM8A6d32RYHVuAWrEO+3zCnRAq6yiUMwhVZpT0tM95eUNKE+/NzA==","signatures":[{"sig":"MEUCIQDIbUJoNkrtXLfIiUTNF2tNPw//yrnJUfH6JuGpe2SgtQIgXSf87i8HPOLllRKGRqqUsqvs84CYza82+67oNxw3Gjs=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1920267,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfavSgCRA9TVsSAnZWagAA4BsP/j7UtGUlHAt8k7CqurrA\ngkVxbclO/bUh212hxRCcYK0CMldjSEqp/qSX71n76Qf7lHQgDS6gS0EW5FRV\nNyE/Co9OUR1HxZp9m0Wg4zR7RdqRTA7HI0uE+8TpMPhtR9k4RmEq4ZsObAIl\n8TYrtbycXPBRrV2XySBRWtSbHrMWBulBGwi8k1BS2rNjDtamUTdmeagM4r3a\nTOdWFfr/pxH4SeHIuhvPC5vLTHFd+oUxJwga1IRDzuuQQymG1JgcX0ctgAnQ\nzLAbY/G1BYXaID9SWeCtDWFDRhSvH2bwVAOPxyFHOzFVIXInr4xhOehBvtWX\netTsnuSi+GchCxvxThbVCcxH59MYPEMWAgqlZ0boED4xsOS5pfcqqfb6YXAo\ndygWWvUFgNfWloHzpJtHSTkj2RwXTxCTgIE49xiCS08o4uf+XNr9Q9vK7X5m\nY8KGf4BYRiQSVFXxXY+EMPeXlCI1pxOMj0i/5GDty14zjm1uivlWAenZLxFy\nBZg/BGlMGsjsg5Xymsn69+tWuooQ9pboVqKxpCIva3SreVOiyYSl32BdYXJb\n5OE9yKTZpifQ+hYSbdWZzKh3sToe5qIEkUl40OODhqk4dC4ig5mwj1U5fjG4\nMNscp5ndgTVGXrzJwwYFlD0KkUjEP4OM83BWbSVlCYCmT+XeCyCJLEvxjioo\ni7mz\r\n=OsKF\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20200923.7e94eb2.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.6","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.18.3","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20200923.7e94eb2_1600844959505_0.45301880168113184","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201003.a485403":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201003.a485403","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201003.a485403","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"4120ef8780904d63932e0f22bb2eaaa8a9cc2338","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201003.a485403.tgz","fileCount":360,"integrity":"sha512-O88kUtkGOcdNA8oipsgxWmrlAspd+G/8f6Wy0sn6weEq75b3/iQF1/ik/st1SeCsRxhXWLw8015BQjhnfmCyeg==","signatures":[{"sig":"MEQCIC6LYa9HSKpidCuJJh+ALAT/LytukFMNkXTwSvraZd8aAiA+4jF6hst4xqWBZISOasz/6YZ/6O/x471MIYoKLYYVmg==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1920267,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfeCMyCRA9TVsSAnZWagAA11EP/iCrgAJvOUJoPl3vv0Um\nIF60fw5wQ1ZdJD44lyUIzz9P1x3mpaTSWMLsj+ry5I/xs/SihjewHVgh2e8l\nmi5sPb61Pt/ZdAEQJLNVtM4MPk1AunqYqWvdGv2SoE1KI3lEpO8cOMr0K1ec\n3ss2dwzhmgor5OwEs3Icm3BfGGn+p7qnigqGtntYdRxB/miRVcjjfgj+5IO/\nMODQ085JsIRChrExUlp0Uk5a7QgRKX8JQSst21N68ldvpyNqqazEkfAEMKMx\nyjbTzS2JIt0jHaKZ1PASmzPuwHl1H5ls6rHMxpEoxJhSm2dQ9c6CcpqfeYti\ncWOYh5SeR7yAj2VPvIJv8HjUsMSgDI/Ficz7wKJMm+oViYZh2uZ68hfzLE4I\nLppbsNqxm/9HFjXK1QqZ/hWPmuy9Ul4iF3rcy9zAmGCZ4cjyv1PQyw6sD0u3\ntMgxHMmpRqwjMtOi6M0p4x1o1U2agu2m3Gu38wJ/Zwbv5yJFa1/Stx3ANQ4K\nn+CG3oWzLxa0djBchm64Xdt8kJJ2sdPjfIfED2slgfeMIp1W4EcXYLiSICDD\nAbtKldiBY9lFBiGNGM9UMQcKv9yTAb/UhIkTx1xrstogoe2QcRTVdG0krCE9\n3hhz65/vU5mpy3gGSPMyjQXTke3Yo6Q8B7UpGq2SfUuaE6G/Q52eczI7l9Zq\nKs/D\r\n=h5yt\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201003.a485403.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.6","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.18.4","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201003.a485403_1601708849487_0.9663329666271698","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201006.29a8107":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201006.29a8107","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201006.29a8107","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"ad54e0693d7f08aa8c6d248c54def2e066fb6605","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201006.29a8107.tgz","fileCount":360,"integrity":"sha512-YvvFNP91eF6/j879kGNxY/T+HzlISfAa4ZilvU7cjQPoynYcH++CCE/s61DnK6snomxqioWKrkQRTRJRv+3mhg==","signatures":[{"sig":"MEUCIQDOLVRr1mZIIh3Ldtys0r3Uw2jqNvMr0k8YtlyhRn6u3gIgBFSG17gmb07KeeTYJleyYAq7iTqf7/Dg4n/Kc3FVzcE=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1932151,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJffBf0CRA9TVsSAnZWagAA2VcP/iXhyoWxMZg31MHBY+Wn\nq3mHoBo3ckMIhwKM7yn5O3MxhaF8j/4QPeS8o1n4ABp41+ZIYfy89qHeDl7Z\no49bqevu7QuPILgf/uJMyI70TkEfg/m2KXDfBziKeH95mj0SuH4UpdZ5kti/\nf8x3v0EPM0KJ0hhelm0Ye/gfxC+KNQzhkm3UB5K/6wQCpnngP8wDlCsmeTYv\n10VrS45eXsWUCgeKq1CXs4XGzjOdjCYpRAwOI/oBFjct0FusmqQvwuRMWz7G\n2jD+deh6zJFjopcpK4/yWdUN1KZyUH7OdRtejV4SA5d+sNt7dWJFEqWJ6Xnm\nKaEYz4uGU9CJcLLO+SuJBUVS7Hst2tP5kDK3ARi4D4vPpd+XEFYqXVTlovGW\ns3gH+JxFSMWs+0PA99d68Jee5xPb1rsFmLc0Wiccu5eVforpWkjjX4cHJbZj\nKU/y/k3Y9QW67fBz93BktXtfgy0i3AetySqDcIO5/J7MHV9VbZek/pNq/6mZ\n03r3VTYdNyk/A5wxgx8NR8KZ9QXhukoTYgHdTR4CwDWZ8jekY+hr41IbFqjg\nNhgmVtwMkc//MTKQy8FfES2EngajSZlsuuFj+U56rc5K33ouG7Nic+C3LkTd\n7BO0tN7H1bTjj85pO+H2w7SNjMKEwzr+Y09WPf4AGVurbRKxyiiMdI9tPH+Z\n/xrF\r\n=zQvG\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201006.29a8107.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.6","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.18.4","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201006.29a8107_1601968115550_0.9196234437449056","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201008.9bc8859":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201008.9bc8859","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201008.9bc8859","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"4aca4476981d1ce7b15e06c69caa5f30911c5228","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201008.9bc8859.tgz","fileCount":360,"integrity":"sha512-ACzfuuM4vPXIgPP6dQd3yWAdXXCP0Nafv33dRSkcemaKw6Guef43tHNC8bmkEOpGRuLU7wZfJhmpf4HUOxtylg==","signatures":[{"sig":"MEUCIQD1ykD79FUOayE3UiX9Qm5P+IYfOzFVDUE0Ik2vuT8E0AIgTX355cufFT4I5uLXcgJ+ebaTxv0aCKyzOIzWVulrBdM=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1932151,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJffrs+CRA9TVsSAnZWagAA+BsP/R10G6QocxfCR8qWUtWj\nJPVpf7ABYuoi3LEippNRn5ftoyEXfGb/2W77tc9AlaRHCM527KOzCwpu/EiC\n8ZhC+C522n8iKHR79Jlj9ZG6MsrEjs6f+zRUTN7094LSt5BSbedFtEhWObBo\nHsemACf5ASjZtu7PM46f+q/eHMKg9yzgoK9GaKM3eLMW9tQ3tbUfZOKXfUX7\nhVW8YGEQDDPnj8ckf8ZJ1Ou/a29lJwviZz9P/s+HjQRIVlZIxEDYi5pC3Mcs\nfKcTrhGnltLz0HZbthOqh//39ye0yiRCF57Dipp8HvcLVYGmvkmOqsZXWspr\ncvM4fkNeQIx7tKl/ug+YhyhV1h9fUXC+o3Opkz3GlDXz6sR+mYy3g/h1uMuY\nlt2Zrb0StjtyFkBcXdS/j3n544Yz10W8+U1M3CxUNRgPQ5g9UmHfdSzvdOtG\nMCLQA8n1/fMgPDiSrFYE4KGqMsl789EPJiIXaHRkSL6wRUfyheulRrRbWUW0\nWeGQTQ4X9xQwXDSgW4UBuhvE/wvlR+bwEWUUmk04zfD+9ZA1Sxz/u1tkZyiQ\nCGoArp4gBGOot4VUY3yhKzX9qpjLMdkisw79euz7sk+vRx20s4pHFN99Z9cL\n8tgHcgHjsWcN2kEAD+2rssX+kxCauSrRwOGW9sxTu5JN+rI0XAaWRHL7FQ6P\nnfGO\r\n=PCyX\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201008.9bc8859.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.6","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.18.4","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201008.9bc8859_1602140989587_0.7462705056699088","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201009.5113321":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201009.5113321","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201009.5113321","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"8fcb365660952631a4920057808abd4647fde8f0","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201009.5113321.tgz","fileCount":360,"integrity":"sha512-LnLiIZ6dbqY2Q5G+p1IjyVrVunkculoflPnzkGFsZnDbn1qCdhXJygjOoN1/b/+ETJ9i1RdUy4AOe5YipwVOtw==","signatures":[{"sig":"MEYCIQDDHuuCOrkrJG+/QiXdUXIT0xqk5YdkGIRLfcqQO5DWGgIhAPr+jzBnmcQVwiRGd+9o0epQQjc2Ca+AUhX7TsnlA2/n","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1932151,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfgAyPCRA9TVsSAnZWagAAcggP/2kvIyNEMCg9lA17BD0A\nEZkrLyq/zJ6MTsI2zF92A3/qV1770jGu68h14rkGenjw06fN3bRtqcQcEb0W\ngfEIHkHrbRNH1kX8ZQ10onoPueHXz+TlNC/3+ncJWij58vHJyjwRkIHcptMo\nyKTdhd4iaq41HgBJ3Brh7HoMhKci8D4A/vbI9I9yKbzNYG0z6HngAhnVxf6y\nuQWA3+Yp9ahAxVWSkw+QPhL1sL64oB7DKlz28WwUCVaZxJcCM0bjk2rHX94w\ntgPbH2PLwUWWsV4RS2duigQ+6757td5YXvMQAn9ZKRJKExb4oBOlb9fCrnHi\n0qm9Rl1x8GuF3wRvGGBWd3EUOirRpHpKBFaozAxm/lQqbDjXFuXNYxj+USum\n5bQuVcGe9m9/1HZId0UqbE+fa5rH2I+WW7v338zYorgsYWBN/dm9D2wixzDR\n8W5jBPvfZYuFp3gG9yVrCMxqcqO18zfBwPySHG1JTF+IKWTRO86sj0pJ+OOS\n3s/fJW3JfOa8ppyziDpv+MgsbSh0EEgg/Yh4TDvDpwVKp9tCe4lIk9Wv+zyy\nLZaAVeaklGUGPh7j5m6z4qm+iVHR8/odV1m4S9qd8ZFcZWm9eWHcf7YUcpmj\ncdR4szw+HD1klNlJKvHfuJQ0VcQo9pbvy7RHOMai1xzsz41CnzMsAbl8xmrN\nYrpb\r\n=q4xa\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201009.5113321.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.6","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.18.4","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201009.5113321_1602227342606_0.5384097177271328","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201010.6819330":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201010.6819330","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201010.6819330","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"bb309178face61cc96532894f0a7e07083de438d","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201010.6819330.tgz","fileCount":360,"integrity":"sha512-IzE7VxKnoKjTEU142hwE6ZW/qy4/Rbc6sTjwbY4SF7MYf9KUooLcp7QqSVcZCMAxfMjl8ZiRf5E3JMZTa8F0Yg==","signatures":[{"sig":"MEQCIG5tQBjLjZCQcxYG1vMn9ZHzI25P6oac/dDqq/X0IKo/AiAYNPeI8DW4rAwkEH0Wc5YBFcLUXelVslAYUePAzi9HhQ==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1932151,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfgV5UCRA9TVsSAnZWagAAb3gQAJDKHzzJwNS3fnRRGyZV\nOf1JFYyyLgoy71Wz7zPODRHGn0Sgj6HkVAZUQg6H7rDio4yml/LQAryu1DWW\nQP5X0PbEAr4k2MEywwSe/Bi9BAyEW6KeeDO2ooAPgTbyoJoaW+zJojnEnN4q\n+fKGnAJl1NSCSbzsHZeLpzFfy4hFz0qBqenYDkO90ft55erqr4LB48LOxVdi\nVeZ5qBLXw05nf+e3rb6fDcp2BLaGWGOuC/+6HRy07HMT6sDmz1nWG2MTYGmc\npueZaRreg2pUC9xQAxfGtggCrnmCtFjA/npUkQuYQxL/c8XQbLl5VgSpFNZd\n72wFDzmarKyxz58xU0xB6S1wz/Gg2fVzMYpEHqW1EvS1Wgzu37LyAPaYYSFt\nhx4B7pqQgJx56D7JVI98mrdaMj5jW7yZqx5rb0gFaLNfFCrDiDEAkI0l6IjG\n8Tj+tF+6rabEHAEAFjxQdtquUE8IcOIpi3BYJ5m/KD36D7SpHl7f0QifCSEi\n35wfmiul7XXJT4b9PoyXj0FbpZlbIjcWOVFxFVKM4454ejhh/VVqWSwwAZME\nMICcQyQXR+ZdB5NMm9rRmr4NKmCSQNpl88EQ3x+VEyfP1yZADzt3zYD2u0wE\npXlKmuYUTUJSjXMuvQCLPoMPEGvVoSOX6/WMeRiKO0jK2oU8Nv4q3DFQnc9o\nJ1nN\r\n=Nr0A\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201010.6819330.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.6","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.18.4","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201010.6819330_1602313811788_0.06857886180445649","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201011.d140767":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201011.d140767","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201011.d140767","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"137b3bdef7862d73ff2e9d1f2ab41f088a80eb11","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201011.d140767.tgz","fileCount":360,"integrity":"sha512-FS5LCP2KakJ77z5k03chgGL8mFFLnm94mAL1Rg5oj7T9v4eMaP71hTpWtBWIO3KIsFNXuV6yD7xlczziolAX7g==","signatures":[{"sig":"MEUCIA1+Zs9+yxGvNiPRSiKqeaXVxxQ3uBxeDn5kzMM5lNHmAiEAgQTZFdU2r3qAXw2kuLnlu5/jYvUMmawtUP0g4wBjOcA=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1932151,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfgq+RCRA9TVsSAnZWagAAJgUP/Ayphy2fpG+8XOAbSP4D\nfiodu4FjGRgYVNDVtoWYLe8HwTo60WHXq9oJXZh1KVQBJ26PTC9OVw5ua+5E\n/N4dwe1pExb5VhVBqU9PNMf5UlhyyfmaqRuH/O3LuADyYw571pl9LMQUfELR\nWqF+0IzGHN/bLxNNCjk1xx1+B5riWXvmItq85lckK4OrH3m3wnLQ1A2dldjf\nJF7Bk4Y26exR/nx8XEdWXYGlRSJbiOrl9WxQzaPnSfMdZu3t+eGeHAySRyQl\ngz3b5BuT8E0rSwbMKKDGTh24bRCIzlfPjwz6A1IVXSyzRBVlXswdcYeRyF3s\nSPJEDM5QtjoZB9BzxzeHJAkY3aFv4bI27e5tLkn+BpYmodQgk1pcrWgwJIBh\n6VQTt4vdQbbtQVPjs9MsU+dRKbu81NUlbP6liwmXRtzBeJobx2RVjhRiDosl\ngMYK1emxP1Ek5NOrpb20Ule7Z/whvuAz8S7WuN8TJaMNaIglOtNlUSLsEa07\nFOjCDqVnvQwyQTjFoxEGFLg7bDVRa2Bwd1/7YcJSU2yg6/4ej6ewg6zu9zaL\n8CkIRzUIdxefwA0Lr97CwpyDIUH2EthWh2T0lnTcVNfVdTZ2xgZ6nxSU0R0h\nR7Ob58juaYjVJ5dEV145oEtHoUIzDJzepVyVQ2Yc2Wn1OwHW27fxI2peD/rz\nLAGq\r\n=CuBu\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201011.d140767.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.6","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.18.4","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201011.d140767_1602400144271_0.4218243630270393","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201013.20d7917":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201013.20d7917","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201013.20d7917","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"1e9d272d2f2746887617df8ed42980cf2c52c9f9","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201013.20d7917.tgz","fileCount":360,"integrity":"sha512-nyyZBEPHfvLpIFRrcDczOwVqJn5qyqCPz8uGCVHO6He5m1xjmuXPH9mKL/nOAkO/THvBzMMI3vCP3iK7h43SCw==","signatures":[{"sig":"MEUCIQD0C4z7kJU+rQichaiQSstnUjS6SvUOiZT/0gq4uao95AIgPX32nwc4BOb1RlJJjd11t8SrpX2mi9Owqux0IRNOLLM=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1932151,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfhVWSCRA9TVsSAnZWagAA9s0QAJnqQTVhlS6kDFFBjtDZ\nb+rO6cihZGNX0f7pmlJZoR2TR2Zm1xhAf1Vvo/NZia1o2/Pdx5x7qcm84f/t\n20gf/ct7iLV+npBf6KOEY5xgVkSj00iM8uQ6V9zDLx7af4Y+qXp2vgBxnjv4\nxCMd8lc/nXa18fDxNyBxnm7KaLfvG1Ho422RyQZ9A6dCdwNl/BMIttw/lkw9\n/k3J8zzhmq+33zj20VU7bJR+S62wscxvvNlPE7270MHIZi1CSLvJAjL2/pHt\n5aIflAeD8G0yL9Wn025opFKR5yWt6SXkwidQHG/BKDws1I1QXEtG6WfASL5a\n2yGIMcSP17qYAJiHWOpO+GcIFKV0PFLeVsoYpuWshdR1hhC1IJjckq19vlrQ\nde86fYCGMouiggXBOB27F4yac9IQQ5Rzxc2zm8FlFk9bqoQcy/XDvdDKpdqr\nxeLg/IvN3omC4/agy7AuJmf+BHQ+BCcyPkISbylLXU7jxhNrqZgCWUF+bDVV\nZtJHjy3IiFgHjPRuyJVFGGNP95D86HdN702+b3pmkw/FMdkP5SKY0Kh11ooa\nhijTaa1dRV0Yho3FVVliYUEkUqL+sfQ1yV/D36uU2z3cN5R5GbDYGDppEBiv\nbhLNbYJs8Cx5Zv8X4wkT6Y5E73U0EdkMGydEOX4Nz7c8Hh1uhEZ6AgWUql7X\n9FU5\r\n=nguY\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201013.20d7917.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.6","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.18.4","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201013.20d7917_1602573713341_0.7430390017369013","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201014.ce6aa6c":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201014.ce6aa6c","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201014.ce6aa6c","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"ffbe8e9e0d87cf8a68eb8d3fceb1984a9ea52caf","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201014.ce6aa6c.tgz","fileCount":366,"integrity":"sha512-MX+M69fI4+4HZuRIKZgpxXKNMMi9meUWJHkYjuroL0YwwpW168FWrUTNL/qFiJZltzkpdbcUDePSK9RIFGagsA==","signatures":[{"sig":"MEUCIQDuPlDAGovT3v4yV0OIRK8hTCW2C8NGv3hSZjD+GBbJ+gIgdbF4C3iXiX61Z9S8YsXrjBqfXez0FgtvhYcTxsYxnr4=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1941076,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfhqSGCRA9TVsSAnZWagAAFlwP+gInPIvBbwxVKgyedCg1\n9K6/ocoH3hofcm9Z+FDzhyeEopHyHyWiIf+jlbDFjjitiv/pPsgLrWhtZlN+\n7K6nzT+PKC3d35EVOBL6FgK6iIo52899SNkXQa/4hbzoBvd0HoQAetNBg3T3\nsiZNx3GyJxBAJE2HkNVWExDuOy7EKpOpZtH1reMx3F4KvF5ir+qqrY0RoVFw\nRz1yC1BE9U6PjsMBBsnijpj24FiWfJX8Sq7hyuhdTTWZORU9lclFPqAvHl8g\ntfH3nxPS3i/L4W5GNIV/OCqvT8gybVEJrd5JjxVr89wIMMmZ2YSkEdmcGcZB\noDemRayL9t5Q/l96WAvJVS/4fIyK+abF1V9npKyKPvJgnxMBRJCFU0fO9aNy\nSCC3+WQ1RfKEeFmk3r6tUv7bcNvONvQQe4q0ASefTNZUQH9rgTA33t1Rc3W8\nq5sS6nOIcAp6qSX89HOFMo8K14z1qlDR3yudxVytnjq1xPaWd8cQwUP8beVb\nn4l0vOmTWDMkWYVHb8B6igzemdFVU4k+7PtUU0u/dCOpqFjsIVa6dI+3Ofdd\nwfijjiOvX/yKLEABqwSf+KvrmQpkzETCxid7Y8EaULYwScGDfR8NHwmKowIx\nnwZLZ8dUUxjVQvGE7ErAaVDJN/GbEnaaySg3iGFIPtk5JNaAdUL/N978H1zC\niwIO\r\n=SRJk\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201014.ce6aa6c.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.6","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.18.4","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201014.ce6aa6c_1602659461418_0.430819003932287","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201015.008a3a4":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201015.008a3a4","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201015.008a3a4","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"a80b68b8a123653ae52e35cde25cb2d1d1b98390","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201015.008a3a4.tgz","fileCount":366,"integrity":"sha512-EN1hdAKSh8siN4UDuhMsM5QJtVlAj5X6peiXB8z18rPNL7oyGUFzgEfUoQ9ZqfWpRUasz1Zj8sAyoFS76NpkGQ==","signatures":[{"sig":"MEYCIQCWiw0FGX1L08Tb2RctWM7keSA7TvoP9pdJ8nw2sUSHNQIhAJ3zugwqBKMmFDcBSXA+xgwlCheUSKaNXk8yiQocuqFM","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1941076,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfh/eECRA9TVsSAnZWagAAjbYQAJfd6hy/C715l1QlFZgN\neuqJuRdrOlvdVdW6QKc9C1ANGp6z0DmzbCrqJtObK/hAELE63IfPSW34FRBT\nfsSqLTM4LoYRh9rZTdDwxoNI0MeL44tMqdnJGGOQx0b/Pp3l1yJUA2tbZ5WB\nLJcMxvLzzOok3W0dC0vABMZILzqlVUtRuMDv1pedZNlmhtEXC7aoib9oMdvj\nkl5avl2cV10e45Ae/cBxPeIQM+NNTuaxSSB3ClzUKvHxsItLyGzfrN7AYC3/\nQUVuZQQIoc/6nYQ2JjilMmAfdJZBRNr3KH2KtVwByj1/ojZgPIyyyI9QirNd\nc4iPQo6ZsXJoHWxEougAcJTGmrw2Xh2wHnuBrA1QmerCuUASl4BjWvpJ/zm4\nBC9s/nJ8aYsjvYEkwSxjxinpE7UMbwJe1Bk8qGhjKu3nA9jafXUItOuPIRSA\nSbHG6hsVSjIexDos1JFayq6L0fX2Uq/P06GWk2XjA+JxnGiuZH2iURmXZ+Xf\n0WZicxgAfT+Q7NIuKfeCnSkG8m/U0LODpFrDfnO5yx+mGvlEkbQKal9qQEOY\n1WytVtqUHArEsf7t+cOn9mWFPboDOonL65LEz3VbgYCPiiUSLfvP0Tdx3NBH\nMyzIZK8qFG/72yaviC1NsYkIXusrQ+1mU9unVBhFpi/gHxpnRZi2RM0/p12Y\nPDtT\r\n=pxxm\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201015.008a3a4.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.6","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.18.4","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201015.008a3a4_1602746244272_0.45429103659156933","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201016.393c6b2":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201016.393c6b2","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201016.393c6b2","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"7dab414752f8711fed37ae5625c38fdd0192eddb","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201016.393c6b2.tgz","fileCount":366,"integrity":"sha512-pR1vVy17riuTos33IvQfDCmGG8xFmNoPUEb376kXITTtugNWVXgsD62P+thk0EAeegr7lCohNk+Zy3rfJ9lgvQ==","signatures":[{"sig":"MEYCIQDC5kZGlwjd1qyv+s8fVTlMKdbYdXoLNMdvcvh4flc7DwIhAOs1evdihqA/7He1BxlBcWeyidMvmEp14K5moeFZGLP8","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1941076,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfiUk0CRA9TVsSAnZWagAAr6QP/1Xo3ad1U4Mui0Hjc1gb\nIdK/cYI76yh7QR95hqpoE5woQV5YAkdPlfG3bs5sCArI3E6z03qYfGOhHNWb\nVX4kzBrUQy24mLRE1tS4mqS4VQWH+IU9MvdFmeIPQ9G7WziQihN2O8rGTJwU\nTQMYYOCzKN64lrK0jI1JOQKPXl+E33biXYqODXai6+wHKx5X0H2vy7SmIaSY\nul3zjkP7jg/vkYfG62nHKIKlD9m5Kz16jRgdLU8pz5rcSzd3YqG6u8po9fC/\nclc73ypuXNI0gLyjpI2PrDrkK6BD0EieoWuNj4TCmdG78hVP5guJFd82Fmwe\nGMl9Wv0eRoPXpwQVJAURqv7hUMpnXiw/a+HWFSXonza0fZ+Czb4C9jE7f/c9\nNnne8MYEu7vZeXTYGEfwLSiwZPuzzp2YIjb+XdVHA0ZcAqbyPaVtxMdditae\nZpulitF95rVveMx3GXfD3ClLsc+DTuxRLmbxsY6skmYZMfdJZXlbikcmTft0\nhB4nhYXhKvK6gHiMLAtnJTX+1+XXfNokHKkwRAu6jUXzzvGlyrz/IKMpOccl\nOQCAbSjcbXXTd/4YMjCGHZCVa2P61QCJrl1DgjNCTu7zJKvDDeoxRQ9ogMhI\nn/r5BIQ7rdilhMAY4W2kvPWePZ65bMfYZaGWOJFuM3DdWEww6djnrDHbPuZv\ni6HL\r\n=/gDP\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201016.393c6b2.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.6","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.18.4","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201016.393c6b2_1602832691377_0.5694734332889011","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201022.4350e73":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201022.4350e73","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201022.4350e73","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"dec6466ef2e4fadbd47ac29bedeed0ae2cf5367a","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201022.4350e73.tgz","fileCount":366,"integrity":"sha512-ydGLZrYOFh8p8rEVXhsuQX3eda6JkM1v4uDZWdgx7uTeeyImGj578h1MgH20mlKrZBIfs6p5qde+w8exmkBPhw==","signatures":[{"sig":"MEUCICb2oN3IqqLwz2OGWv5ZxEsioY9JtNemH5Oc/ZUiMhElAiEAkqU+fe9+soNvYAZzWDRPER/W8kwjwdAjOmU8+/UPMfw=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1941076,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfkS/qCRA9TVsSAnZWagAAC50P/ja1mWR1kTHf1iVQfpyt\nWGGdIIdN6WdXDYknnjJFoNxCd8GmrbX0CCW9rCNLjJWfcRyeetieaghwWdO3\n7sQlcj6wRFd9JVW08vyaB7Z1fEd4JDCEbtacubtIWHyF5+ygcVqGAhgJwHO7\n1dfR7jJ4jXRiwQuxHDTBzC+6wmFQgTYuwSoL5WMADnXhe3yB5hDXzC0HgIQS\nSnS/DJ0F0H33FcMhkCA9adOiyw6avA/Jgni1Biqr2O0lb4KX9vBlIIrT1KTz\n4TQEY7HxytW08Igx+z7EEfTGrU3SJutfMfGTdxbjiYqsVsfXaBXYLj1Y3iCX\n9DN4y97N0dfsq7CAx4xPtCw66tfGK6Jzdu98mtpIgC+uppu3YkzrppfzyXQD\nF0sWPrbMI3oJ6ZMVds8zE6QZI1DZ3BFFJSyvXv1vw0DP+5Ti5PnktYEiYFH4\ns+yysUhLLDpU0QkjyFTij7sCZn62tPT0oLcf+viiSjpM8+ExdyUmRb4OxImw\n8wP6gwutSTZJBNJ3rk4i0bT/WlwG+MYO7YOGN8sjbyrKwDHioJsg3PEar2bF\nVVrIZ3zRc2ST8ywxxB8OIOReAPcUHPnJrcowwO3rLjRmt2fsNhOqa4cTT2cZ\n/tZKDgkX6eDA4YZUCS/wTAMkucHKdPErsgKtsCo9dZstKjfz8fOZBstY8xRq\nWIX1\r\n=3ltC\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201022.4350e73.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201022.4350e73_1603350506209_0.8305879077431524","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201023.67f9a0d":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201023.67f9a0d","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201023.67f9a0d","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"96c7ebf27252121d2f7ced5305bbffd3de79582f","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201023.67f9a0d.tgz","fileCount":366,"integrity":"sha512-aZdU6vbjP9DC0n/wq4x4vNOYaEoR3XgVe7xSEJp5BXCkUUUtneFttduNXwKT4ci31aFnCF90hZj/z01jx9bt8Q==","signatures":[{"sig":"MEUCIQDm3Rpn5eFDJNl3azK6g68X1y615z1NJmm+38fCXaPoUAIgeB7J2vpJaulYHvM38/1EOv+7QtFWk7XTVE3hgCVPP6Y=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1941076,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfkoHMCRA9TVsSAnZWagAAPvIQAKC5iFc09kv8uHvbkZQ6\nwNuKuvc0TwEjVTa6YK9zXJ7wlRg9RDrlyqzx+m6hbMph/Cq2kPuihUX647W1\nBMgVVuHR+duRUAuGmcjZ33pQukozDLt8vzynFvRLeME37W3XOrpTSED4THx1\nUjjAx3WTM5rOuXsm0GpZZBJhSstseSywANA4ICc+NZS90nm+SFatx1Q2LF83\nAfzeXwmv+6RO9/GxxlZQ0xz+fBb45QVvyf9xv7lE2Wz8SZVROLNbKMoUbGuA\nCdd19HqtAqKTLWDWFKGD4GqoVKMv5fQfSm4xsuZyuD0CIcb2oZYyGClxsyun\nVyv8FiBp9WD/0E5W0cZvJMbqy1av7wIJ153bOp8Y0/H/6OXTEdWqJPPH4FCE\nkr8nrUsRA0CQrN+d16DpqK3gjIl2yC2D3xM76mWEngq6xvt4crld+Fc5c+Nw\nFtekzJoI0lefXqjxJXqDCK7exPGhUSzXtyauXMsIAOtnntc2ZTWySrG5ut+5\n6RJBOLRwoWRLxa96kRLVxhHjYANFEMIElqVkrcZraAlb5WrECfYSQtNgWhlV\nRBPxSF5z9LsoJa6gyY+QZWATQ6soEC50k2Yq1dak/wdMsq5/LDaNYaEbWTeM\nWgc9HhMB6DFwAsO6fe7HW7tRohYuSG19CtF0HvxH5tT2qQxK6bjzQeT5JgRa\nDXw+\r\n=2jU0\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201023.67f9a0d.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201023.67f9a0d_1603437003897_0.32821268110566804","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201024.23949ae":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201024.23949ae","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201024.23949ae","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"5590af6b66c0836dc4d805e8ef7460e9fff639e2","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201024.23949ae.tgz","fileCount":366,"integrity":"sha512-EZHAOCA2L9nnCgKWAqL1BVW3E8ZDk2nH8ztNupouJQ973r9V3sSHoJ6+aYfgLsLfGRpGdvWikbVhkc2Z0Jd5HQ==","signatures":[{"sig":"MEUCIFqBb6cZF3jTN/U84an3iLOTDmBtYmHCQyGPmcb6TZkbAiEA4ZNwtDI9kpWUFeOYMoFwEdpj6VoHoX9fJMQvrzkTTh0=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1941076,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfk9MICRA9TVsSAnZWagAAFeUP/RyGDwNptdzpxNMoIgvC\nIgbMcE/m6NKw6+KHTSuGqKb4ySXA26MjpiMS5ohOoOkRB3mKYmd9f7TyBYQm\nmJ4aiK2NNSx1+JXNJeEcmnEiRBRZlCfVNe5HgU7Df99mhCQlj6e+C7af45oH\nYKVw8Hi//uZltXX8F4bGk43I4YTHUgurjugoEzMcvB/gsbdkjNZ1McHNzxFE\nTRnQlpxYT6/ziLcTDg5vbdpd7JOdvwJqHAs/oy/cVWDZgR2eMbGXvx6fZmnV\nQPxqh4YIttwVZjY5t98Ldw/6NmJ+uQJGjSEAjcONuEEGRrSNgGNWtOkEhxxh\nfAHXE8OXiHWNkPCeTQGGn0ag16Sl91dqIzeo8zLM7tCtQqEUlBHBdF4sQZoP\ncwbdgA6JYWFmSKbBqMVjxK0Lu/CPiNIv/mOSq5DFmZnR/dp+/ktEUhcy1FxJ\n4yuVOpOeTnp28xdv6EKyYUi2xv5xoCxDrP623JR4S8WYe68QNuscG6/6Dlld\n4sUxNZKRpAC1u4pFXiQmWYYhr/TRqh1suTXc+hap0W+8pY5CoBelz3Qm388w\ndV36xE+6IPYRa7ozBktqwNzT7m9ZAitdaj8gnES5E8Muw/043yn8R6c3DHFN\ngFAdvAAt6HCP/07oRYufc0RTHjVdecPsj4zDO8HnRjwZZrUjntixwJkMMFW7\nTHvD\r\n=SxCe\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201024.23949ae.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201024.23949ae_1603523335570_0.2389047417525072","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201025.035cdcb":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201025.035cdcb","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201025.035cdcb","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"d8cd0da10cc9a3c0ded6daf251f344d4b7ea86e4","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201025.035cdcb.tgz","fileCount":366,"integrity":"sha512-64w4wgy0ATJ2qew2+Lw/ADPkz/OPHJ3YyBfVnjzN2xLbXwoow9m836EQSIS5KmcMpAi+UZyuiMotnHO/fUyFuQ==","signatures":[{"sig":"MEUCIGIHYqFbNrY9kUWpfh/Ec0dXTesS3cVPgfUZF19XI8hEAiEA/I8O3nk9fAaN184hcnsAjfxc3GKIn6WheKmjjPcr7XM=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1947934,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJflSSfCRA9TVsSAnZWagAAZjIP/Ah2W0AdNHhXyNgo4xIL\npOyzMWWdT/puQBOZY2e41yFIczXvYhnCZSWuzKC/wqWiY7x/0FqB3+JJXXzP\neLGg5C2ZYo/JSDUkmrhNr1XqUFxXxF/gao0xE1fJieC3e9/Gp5yNdO6m+V0S\nG7K3Pqn5ax1KQRt9PPOVStN7DA6LV/2v5z+FvxUPV2//RV0HxonQDBobwpRk\nNLtcoSY2vaOAg0jd9PT0utE5A6xbXksVjYWlrC1K2sjIDwCKiNTB1ZwOYqym\nWfckkr8E6ZzLzbrK2AX3tE9bU7hc4LBETbEoPqH9EpSHm1UKQOH/5lQm+/oV\ncwLLcMcHw0KKnRALjdIQxWNtepIZVUnibvloEDTLo6mXffSx7f5fDo2LIvLy\nTD55GCUHYFVHYscujZt4EYEm2YED/wGLnoJSVo0uwy0e+TT75BPU3r926HAo\nwaWjhnC8WKvSnrKGGu3GrC/+7Oc1Z8NauWwz2gQnYbpmR6Ga+glv0kYspOfA\nBkwvBWQ0v9zx9GS+xCGw8nvrFIcfxMXXIQGcJO8wow0HuWpwQV6KHI//MnlP\nm6plJi7dql0WSTaJrbaq/oyoDtQ+0+B0wM2SYN5LwikopnsTwPLCY6VWpVnH\ni70EUs1pWJY45U+1H5jryscxEYy8vHGoHU6yjYb/l/D3bVk2tevRXU6aKPWQ\nPYvA\r\n=gd6Q\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201025.035cdcb.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201025.035cdcb_1603609758828_0.16675254206477574","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201026.d8a7aef":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201026.d8a7aef","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201026.d8a7aef","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"66122608af5fa2b44f2fc249f1183ba8a6b6b4ca","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201026.d8a7aef.tgz","fileCount":369,"integrity":"sha512-bpuKOO0SBaSCEdYLxF5ZK+zr906UlJao3xjf6qpo5jDaTdSvDpnwPd5Jcoj9S2XWHGkpwmDX6qLYhroW4Pqg5g==","signatures":[{"sig":"MEUCIQCugsvvinlkd+JvIdJBUyDP9MhS3IbOpB6BchDoq37RegIgHLUQ/uTEpB8QAZKV68kDCK6T2rksyX5K28w8XPfcKaE=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1955308,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJflnYoCRA9TVsSAnZWagAACwoQAIyTSjfievz+BWqEYuZd\nZDbp6ZkMJI4yunVM9TEgLq6LL5DdkXtovjRJ0qAO5/F/y7bybZQI8iDLZm+f\ntpSA1uKILk/bD9T4cAheGspLM8Aw6DUi0v8nh8XZKfb3Is6IIeZlfSlV6dit\nKvt8iVXBQt9DGhXig/18BFei79hSdjvjyCo0idqw1z6ngOzf9LBr7KQqNARa\n0NKSzYB3FBiGBnN76bKpFc3VSjFPa0yQVuQ6UFSiuXEqhAPkIBtWfEFm++up\n2NRyFKT8tedE/fAFBuoGv83WRQMj+OMnMvafJ7ZgxopU25rgupXwUMYuREaT\nNIy8V4ztDCc0XLg1hFuLe//h0ns6pH1pH28JXGxxGtEEnUYzMYMN60gr9DQ1\nTA1/kJeD6kc9d0yl462vpR7ePnXVHjtmWoX6Jzxgx1Gi0R1FqZgEuGbcX+MF\nvRY+4qupKt0b8dtdgOBjR1JTeGLxQr+8YXAhzDahLSB8r7IEO02Z5HnbZUax\nJY1NN28EAUakdnHXfLgjpIfk9Oo5IbEfprgF8gCYWkt3DjV9TSN/z1G1ZKp0\nCdgAeW8WSZfMRhj1woDh8OoCGwrpn7oCN2auzo0lx3BSjRpBhxRfUraH3ztf\n7JSCjMYJbUwQbyUC4CbWf0RpCz+hpMJCP1kkuAzgND+SQg8IEqLdGNK7ijpo\n9PjF\r\n=jP8w\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201026.d8a7aef.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201026.d8a7aef_1603696167623_0.7842960019973895","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201027.744fa02":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201027.744fa02","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201027.744fa02","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"ea3ff88a835601a188f56baa33038df517ac12b4","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201027.744fa02.tgz","fileCount":369,"integrity":"sha512-B8kQzxUDK9y6FrtgUANi7fGj+zyRvvbufwoZlLCVPUxL6XT2kOduGGfnfoITPxtSWSVFbolwZEv6EEM2IzyXJQ==","signatures":[{"sig":"MEUCIBNIguEkJwSqp6y+KXF5O1l5cdU0pBee6owFj7b4nACYAiEA5LOSYEGwXegIaK3Pcu0DmcQasXja/N6SoiKt1IvFqUQ=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":1955308,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfl8etCRA9TVsSAnZWagAA43EQAJdqj5bMZRTAPIquoVMX\nQGooRp+gAKiY0Z4A4/4Hqmt5HRpE9uoM5W9+7FFPORCJ8DB4JLA/Bl/6DBq2\n7AOl65PSBlyPIRs9eK5DvRHdwOfQgZYsLI/v0f9f17Btrwk5nw4uMIwCRzSx\nNUoiaRaEbzL34zZk3TmGs7bKHs2nbbxuo9zvOeucOc+1Mz1YTn1v6Grr71JA\nhel787j3aj590INhZTklMcVPhlxd20JCNTFZbyD1LYfUB/OQvcMw3KO/yjt/\nNvXGw3zV0f5KV5ueqqOBk8s78Xr754d8IqEEBMNZYDihf4NRxC/j49yKJKC0\noBvndDlMLkvGdFu6CW5npoiQTxYX7Jmp+SHOHRWlOQT/h01hWBTMvpoXyi/G\nUEX4OnLRvRVU8/bPEaa0Y9jT2yrVrmZd4dHecK6/gdyXAi42OuR6W0b8noUF\nXDVvbgN1sdioACLgu993jQDfUrZEbLVKFCUfM8BNSKWdWUM32vp+HamfLNTE\n39ldIvT7+v2oSzIx5cEZeQseb765DjI7yZ45rGRMIVhGAOXE2hqW71McmpeD\nbhZjxQCrMguATipxqqzlpc2RhCdcMJeEoDhB+ImlsH+8KaGpx7RUrZcCxG80\nFqYbgv1LiBbksGipYrNUNmMcbsCV3D28we1mo4SMY8P6A6hiftBx78wCni9g\nQ35F\r\n=0mg5\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201027.744fa02.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201027.744fa02_1603782572893_0.09666689398667927","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201028.1b0cde3":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201028.1b0cde3","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201028.1b0cde3","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"af160c959ee9c7be67f62e41aee923cbeddf1b27","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201028.1b0cde3.tgz","fileCount":393,"integrity":"sha512-0SLulmID6itr6O1MLzMIF0J/BjmCXQUmmzuB6sdG7SK8S3sSkZE7bsImrqWUDDI8w0LocZX8K/JKZ93Xt2fFaA==","signatures":[{"sig":"MEQCIGxtPKzf0rX0Sw13Mrp/fuFw1PdV6rGWdQtXN1eVpkB3AiAM1R1zG4lWMG0gIFZA3D6F6RnqU5K34RJDOSWnQYVyIg==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2007067,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfmRktCRA9TVsSAnZWagAAnEcP/1y4zy3OKEhakRLP3KCy\n9Sa2+xyrjC8ZlJ99eIQ8BEB94apW56vSOkD/JsCphR71wpvc9PaX5VrpbQaD\n7+vnlKKLiyDruGRus+VxYZeRIyVYkAGa+wZ3FswTnf90zt96QsmWTlhtpkgi\nIQVP3po8g3Vanc7/X8x6U5f9yPZEBjqsPSpQCxxEE40KQ5fwGhYctV8OnRAG\nUtq6QFHEKVXGXeKMErFih3FSzlC6pQOB/lxJMtriuo3Sd0nY+cWfW77jVhUG\nl0JTyEKigwP/mUHEHfUvlJwHI51Klz5clk/fsFD00ynHjTYk63DTY6hW6SFf\nHlzfCw6I2tDMnZql4ut4UPxKRj/2ZUdur+umtOR6iL9ppauXj1ZHpsGGzzhz\nl1DDAYPZdshfMsLUcxAraXKa4i/KBXJCE0b2JMQlkZk5mdbG3GInGThGb7aL\nQ6wVn2Z/zLWTeqXsHk9gg+SLpigRlK7T+aO/Qk/2TKesjNG6qWZVSq/3Vx89\nPSgcjydYu0Fd2LIBxuTpNL6yQ8Eqsmu1XWuakgE6maMEk85fumK2t1C+URxA\nmw/ZbO768zYpHqa7N4/CbuCCwAyy6NoeV2PY0171cNdcnw0x2m5IsninfD1S\nfiqikj0wpOjpGAqEZ6LGCpa1Hj9wEGE3PFY5mpJMTCOK49qrt5HJqNNSVM4H\nZNRW\r\n=tuRb\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201028.1b0cde3.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201028.1b0cde3_1603868973253_0.7472969117335164","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201029.e864bc8":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201029.e864bc8","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201029.e864bc8","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"63f8e0dcb099355226102a568a04a5100f4d9aef","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201029.e864bc8.tgz","fileCount":396,"integrity":"sha512-bj08DG61cCIGx6asK3nZrmQI9KH/e2YSUvy4HcpLEkkjMYwy4rbrxewxxlDzbw+pwtOz2G1Aio5PHjTdB1S7sQ==","signatures":[{"sig":"MEUCIQCngG/9VhTCPbdlmtondl/pkvcTKtOSZSpw8QzV9VBhZQIgBVirFy0dgbsp803Mrf7vMGu+OiOU4sRr3wa7eC6MP+Y=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2094982,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfmmqECRA9TVsSAnZWagAA5IkP/i+jyl1FyNNNlRIbEnVX\nTZTOGnsGjYVivtMUfNaJm378zDn65R0yMdnyJwN8kAMEQ9rmI4yq45eW+T2v\nVOpfBl0rk+Cq/sjeUPC7XfNzW7Bu7ZHax26+takMZ2DzeLYOcqy3/JZ111FM\nINFtr2aw2eO2G0TjfleQpP9HR/HpO1eNdOrrVKoRm7lmDJ8+UznlVC0yHBUg\nBosVZ7I0/VfIZUosRRSHH5SzdLp7eteUlmAV4rsbIiuJjwSFK7Z0KyRWTONn\nMPPkYFUrBSMwsILBlPjmfOC+d9AyB26RV/OHG1bFbvlOs/9Yony463/AFmtl\n/6+lBsVq7HT36Etwx8/oj8P7N4n79GRwramy6Dwodl/hX/QXdKQdcnB8BKbb\nalZlmBBX4R4J8cgj7eBW7qGq146QjKOkO32LRWbeHji71ge0+LLc47eodLpR\nZ3wQa6g63tMRNFPzv6Nc1XcsM+kXlRx4ln8tcprpjZXhj7T86SR+4KnYWqbX\nZMDg3SdVlJ21WKlXpiQR1yDpS1U//NCYBhwSkDUxE/cfeY/BrozYaC9D9Asb\nHX7VgYG1hVBfTAezN0cw8V8I7BBNn/2P/bhIsPcf+OAWQfFHi8abNtdwc4GT\ngtUU4l/yjH4UkxzLh2cXO/gA5/WI1ENnGHYY0g/tmlF/41g24fKtXg6caIfJ\nFQhz\r\n=d46J\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201029.e864bc8.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201029.e864bc8_1603955331818_0.678482949183066","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201030.881c37e":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201030.881c37e","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201030.881c37e","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"f31af715f6dd61f0cbd0989eae5f050a8dc26bcd","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201030.881c37e.tgz","fileCount":396,"integrity":"sha512-NAOf940+Z2eFKLnuCKuvCh85+BotyAWwGqR4mNuok38YaR7nkx7ZrY+RIfNKF96UPtOHI3CqAD0smJTwyd3nnA==","signatures":[{"sig":"MEUCIFURUZhEWpSJtnw9tbESPMb4RP/0OPNLoNptEqgbsLg5AiEAl7dJZTOHNsI/nkwRpIJ9e1o8SC12FlRo8axohDjVBVg=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2099613,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfm7wUCRA9TVsSAnZWagAApowP/jJ9EKbKABCVgp83JLv0\nB51U2SdoAkArxeG3HGKyPQOMTBS+Xs2bpDL7set1UxpwZc92XoWzI/5pzdZF\nvixuOa+tmgEPDwpgPr5y1bhOsqXlgAugyDj6/L9sbI/IMVsAyj8JIiISeCfG\nyUc88m1jhT9Y1APJzPRbRp/3mUxbEL28YldqVGroKm6CxzYVyfDtbGRQDvQF\nE18LCFlAxJtp8qpmsPkt40Nc9GEs1AEdsV+vsRLNZspiC9jJbmTDt/Skt45z\nGDvFnCMkf+rLRl3ad7z1WEpVdVV8Rvg3cFcr7v0w2mE8iylU1tHnUS1Auc4p\nAVr7bBspHLx9wtkSE1CZFbneQ8GuhTuB6542lutXrgi9nTH9cgAwEthI3HWn\nq2o7uZ5smAi0bTPA1vDFGyfFPcwIEyPfe2RUErXxeJX4LK2LACoJizP2wOKW\nzahjC4BCow09CJ+H27pOxjRxWSwd9GcsS4pAVWquV+i6ToHoZ8Y7aL3Z2e4w\nzNMIsUzH8EgTCMPqUqe4kyVqQY3TN5X/YKZYiSU2UaPv0bD60HGJjVje1Iah\nP2ZalSE/ReQV4nCRlYnsPndJqkdAlvFB9hzHHySzf9r1nYYgAvAZZoTngjXi\ncJVwUTxuphWSxn4Yaon9qA45f8wzmdXov+kdsxwbzA+MPSMpA6XwrNlzOXkF\ntetf\r\n=36Kf\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201030.881c37e.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201030.881c37e_1604041747817_0.03209316590923117","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201031.17d6e9e":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201031.17d6e9e","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201031.17d6e9e","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"1f74c1fe5f93a5c2a42e27727d77442b2c6d6725","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201031.17d6e9e.tgz","fileCount":426,"integrity":"sha512-DUnEiV4+FGyAUY6C9nANq/f3dWAKi/Drf9XWPFz0dyM5jHF9/nKLiGmw7g0AHgQLBjIDyKtSsFgpug5wgWLtSg==","signatures":[{"sig":"MEUCIQCrHrMgrkADi5UQk/9SrCezByqx5hzZ2ecDQ+kigZiv9AIgRUm2PC1FjeGv6FpdhjuIpB8o1M8acaxUPnK1pqu2vhU=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2160261,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfnQ1lCRA9TVsSAnZWagAAGQUP/jeoZqXGzsHbBQI1MO6b\nrCmZuFk0+S1kTB1E4yDMhCJGKJJKr3rsrL8tIogJubIOYI0cyqGmdUCa1RIu\nIl0N28KNAtNipFFGI5Js357ImECi8x5g3ZBwsDUYC5XLFp5aG2dYO+C74ya/\npUXH7N94CV0pd0a2XGDcWRh4hymi+cmi2q5gMwwqEJ9eoYHyFK8umW/wa4xd\nF4BfrJbywt5f4tqfZEBkcSKNWig4ltgp5Ca/vSb/UKbJdeSdO6r/W7i4O0v0\nHj3O6h+u70Dii7dIUN55yFx6/9xSWZZ9cequ5luQkHtE4L+oxiW6u0wqvCRB\nJfqYTncQ08NIRGG/K378dmjcpyEibPnkOpB2+D3AJnNl+R4JPmr36L1307kA\nAib6u06Cv62jtn6CMHwUAu3+MUSNjtaayJSi/gyjlSz5KqcUxmD18KcOlg/k\nzmGWqPi+LC/wQZje4viDqPas8VTBTtDPCKBzCfcINV8IwhgrcILsqXzlQsBX\nCXKVQgdzEJ4f75nO+aCucoRtFw3/uj7XZPRj+BaH30cgV8mVeGYmLNXbJavU\nrl2vvsqxiS8ddMXOzwXgCj+KwdMA4SQzp2k9Ctnv4M0WRcPqdNURwi2eiFxT\nR7QxG1hU40EQCygL7jxCFvXojLSxKIo37pI/vahqUHHrrgbz83fR7/YXc9HF\nL8oo\r\n=HuYJ\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201031.17d6e9e.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201031.17d6e9e_1604128100904_0.9123064342732303","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201101.07f38db":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201101.07f38db","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201101.07f38db","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"89cfbfabfdc9b7fe446bf398686cc022bd3d03a6","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201101.07f38db.tgz","fileCount":426,"integrity":"sha512-acEQlljctOi9uWtpFvZMDgRRU1LNEXgm5PHQXG70APK3cj3+y43QAzPTGgYXqq2GSgt2JRFy6BxWDEkCz9JmLw==","signatures":[{"sig":"MEQCIDN3yxiiee7ErBNKp1KKpzZERuSqagdeVi6mL1/57aTdAiBjUv3bfufHnLVMUbwNw71HVMOZ5vQ+uzT4a5d5MZ/JQQ==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2160261,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfnl8ICRA9TVsSAnZWagAAb0sP/2NMclo+lq1qWM4qtfxa\n7wOEY8Wx0T+V9xasFEGzGL4SLkBEUem2Ne2QNsGNMtYvgnXG+QWeT3Olm2oh\n5TqXR5JXJwgMDzt9E9tWETHYn3KJDIDakq/kkPRqNGRJW9xZVPbDtL1erk66\nBUG5cCEVRt8BL2ZQw54/T/p+JushXyphrza5M3Lxj1P53i7Tu5tLAGKJLmJb\nsCdm7rHpAEQv9NKvcqINxd7PJMRkdQ+7Hnsv6OFvW5ta14HJidqwe+irpbkS\nUV2h4qfQkQTLjLXjgj2tPCpFDpkwODUwh7kUEuhhiQzhXm3ltixHeLaT8f9B\nKjPOzd/iwOlyzajS/tNvj7trgg2cMMueqoRlzpg1bfcunUYf1SANNIlyV/0s\nkLFmf851KrHHvY5vuMD2bccEyrbgwXGtMrESpTVpT6/JanHC20oZkt0GlOEO\nNtezR3LJaxBX4CP2/tjZhOgiT4SD3y5lAiMHeXRuUZYeWZaSJXUGRrP0VwaK\nin0Mtg8ht7BfXF9qoVPPShvsS68Ck2HmOFXHHkdiunTuaWGPe69CmCHXDNS4\nLodPvrx829ti19ISlRu0RJ8jkLSAzF+ulrX+bVY7Vg5KqeWsJ6dXpGDAPoeK\nu7ju1PsG3VcKD9K/8m3Gv30x7SlZeG2p7j47b1+BqFQPjzusKPK240cLtXY4\n8ZpW\r\n=nmHR\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201101.07f38db.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201101.07f38db_1604214535633_0.3775502769100567","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201103.f78e0cf":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201103.f78e0cf","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201103.f78e0cf","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"7f683a0d6b2c96ffb5e2bd0b937105884a4840b5","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201103.f78e0cf.tgz","fileCount":426,"integrity":"sha512-bQw76rdCDRnnpLJmJNwRJMYhfdtznuyzfInvmYWzLHAs/pWU5JlECOXO45+pVhmgDhf9Rm5JlKiaGSVfzCuGhQ==","signatures":[{"sig":"MEUCIQCYpAPD8wBJyk/+wvtzfOiin1SOs1PNv+nDy05lPc4fAwIgVZdGHpRVpxhiDPkDWhGH/NnWTpXNXscc+R0v+ceNx4o=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2160261,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfoQHtCRA9TVsSAnZWagAAM3cQAIpWdhYfgBKUGwXD0rON\nCLtcXjuK7UNkUuVToZxVEozEd0SaOwHqnQzzmUENapGkxjcrWbDTgWbm2LHj\nfiMIE75cl/ezFamHmqBLWFZdMvjGCAD+MvOM5JAdtTe6D70dC74Gx1Ii1W7/\nM2XqsfZrbWTCnsJg2lWgHvnCGk3iKyXYEFbrpjvyPpRGZqX8KbILlylhR0zH\nRHnRTNbrTE7AZDU5HwuwQtG9Kwa4wQGmmElLG+N6QCMtKlly1AR2KwK+YB7d\nZnGl8Y2/KgkABnMNKaVFAY9XGWPk0okeLJL3jbHbgkLUuPn0nHH1mp220Maz\niLuhwM2Degn4omOwKSIUQCRCtSScD3FcHUK32kEBa3ZhFM659IR8E03QPTGC\ntmZqf0v7zM4S7us47ApYGV9oTQCG71OS1KLno8j9hFDN2jFDEAdhSvb3lCl9\nFZsS7bBU4M5AXmAUiE8MECMFl3xiGrGWnbQuh88RpsHW1GHbEqnbYZThUNEr\nPsKexIiRlfiSIjftS4OxDRrrNxfPMjGBw3Z5//GNaRMBW2Hldqe01B1P2dZp\nMw0vnq8kperR5bxS4OBB+9lKNGwU3Q/mhnkHhiKDY7oy1kUVUh4d5cONyyvA\n3W8Maj4HkXb7pUPSpn6uVVTHlxX61UB/z3rqixwnV6g/z9WLCeGO0Lg9uuz1\n//5t\r\n=ywls\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201103.f78e0cf.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201103.f78e0cf_1604387308872_0.8807512057693412","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201104.673f22d":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201104.673f22d","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201104.673f22d","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"b2854d2e13d07222c274865edab6f1f311c77c4f","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201104.673f22d.tgz","fileCount":426,"integrity":"sha512-g5GpUb2PS1Th5qJkmCHWQxR8T1n3mHCROt9dvjH0dI2JmkvsVPUOZeu1/yZTE0vYHjd9d9h9DvTUTUIv5WPFcg==","signatures":[{"sig":"MEUCIE71BALjBmt1cR/WuRRzntoF0g5VTGNsSBSEvajED0r3AiEAhudNNDsxMWBCZsWELtZvm2cvSiL/baUM9LdoNxLD/Bc=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2160261,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfolO3CRA9TVsSAnZWagAAwogP/2OxeJXgZbX6hcWeIukp\nbapKTqgCL1jerHAEVq2TyOQO2XMCzCfZATYfUWDCDVEB+phTdZGErMnf57qS\nkrrPe/QSVUbDFM4AOP5UXYlJPS6ZsoSdstFJVoiz4dcZoISZ8kR+9vn3E2VX\n5sW4DLHX903cIHQxfQg2ipOr7X6tTNHm8WTtEyTTgvnuoXCh8udRDZokCdyA\nrVBS31Gm8dq0T94otyAK8CkN45vs3zGh+4ZMWEpXwELyEF3IzG9rJZOa56c+\nUFkMPntN7m1lXNwVEQ7Yg41j2xKHCdGNAb5fIdwHJV7y+e/RRo1jjldDrYCX\n/gUbV0ggmh6UMHbhs3FonA/smC800UiPhHNvdQZRfl35QQSjC/UJfIxb1JKq\n5XkmmyH3W4XDiqQOklhcj9Kv8ADdq9SvSTxUEIiI51Nz7RI0fzIV8wPYjIIo\niaoDGDNLri9EpbaeeYRYHJIvFesmKXYZoSi23V+a9uJeChZuk9KPiahMfMcp\nKQI6naMvmkDT5x5xgiFGRevMuPtPholneZcqxpLXeXzTuGbAs1cm0gp1crVC\nBOLjgGh5iXZQCIX9Tb/rd6my9I+h0V0hdK1NJjRig0+ivrFqYLASm1JTUz2k\nVtS4xPI2VjM4bvaKyJOSIXik++GaPgxH9SKMbmALw8x3Ex12tKgKVuYYCDmJ\nju8Q\r\n=vH3j\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201104.673f22d.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201104.673f22d_1604473782915_0.4452151919348999","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201105.0cc03df":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201105.0cc03df","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201105.0cc03df","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"def11616cb03212444e934dc17aacfcda731c25e","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201105.0cc03df.tgz","fileCount":426,"integrity":"sha512-m3lVnT8uH0nSGVt8m4Enm/gF1vsp7eqqA/g4NCGGiYfChwIMBjTwErOzITTHn1Cwz4/ztUw3/Bc4tTXuuOjypQ==","signatures":[{"sig":"MEQCIBiHEfAHVfWpd8AABtU9oSZYHLhAxvLc8Tzw3ZH25WKAAiA5LhchyCeM6kcrlqv5kq5bZ8x0TW8WdbPLO0W99TuASA==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2160261,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfo0PVCRA9TVsSAnZWagAAJQ8P/R3E/ClxUJ6XLJM7pcSV\nBb5o/i3rvpDGUzYluCTNP4xwLDODN/w2DvWjjVyaac2rUUQvMvbVt+H2I5R/\nc9HLyFB3MKlodfEVuP0EQP0AOgI/hdHosrNjP3EHmSQkS8HcLYDU/aX2W8RP\n0Nx8x4y/PKfnKRCe0ydjSfxYdkuXK8APeHsKiB26YHJuuqePPtE6OwAf/4/d\nrK9LpVIzgdwrez4kuP3RZCbRMFar17a0c87InLPyPKFsgLgNRn4I06/0U5PV\nB2R5GF+ZNbGlVaSLciWz8QB0Etu4pduWjE2e/taPt1La1md3gEgfmjZEdbeA\nY4ESRMvpsWd+RuNjwMNr4SSYgf8nPx6HmpLR4Gf9rA5PwcUVx6ENkjQPYA5v\n95aQcmJ+ULXTYdwkmjnGUW+3XI0hyNELckrF8dpR+pbXoxtih/758eDKhkB3\n1Ru2DND2LY4h7OnhfANkrpHpaQBSLuLrZf8QPukaKaJMw1/fCiv+0ecB6vz1\niS7oGeXXokvDyu4P19JnQJyqNgdMqArUOI80ghO9lbPxJ8nQ5LAdhwe9YyFE\nJVJvT2rxOd2anKc+rkrHfjzpwjxFS5tVN1W3US7lqWzZDbrDMLUDXmPjIsfn\nB4vwaPy5d/FNvg/EL+IAy+rT15uR/d/7TDGozpYgMT69SxkyffQ3UTBW/yWY\nkYvQ\r\n=Y4rU\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201105.0cc03df.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201105.0cc03df_1604535253106_0.7432522811879338","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201106.87dcae0":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201106.87dcae0","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201106.87dcae0","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"6ac40eed6724e4edf2a8be4ce18783c2ced72922","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201106.87dcae0.tgz","fileCount":426,"integrity":"sha512-uLG9I5HQxaSXU7qJcd6qvz7R36bXYCWxuSWSSbtJ557rI78Gl8Zj8nJdZf2///12njCWuihi6HpzW11DRyHw7A==","signatures":[{"sig":"MEQCICKFO7nkAzm1+2g3XEelnoYz1IIhtwHuALMnYdK9dYgYAiBEtw+BwQIfqv0MhJe6vIgvacilKwckStkdmTLNrFI5GA==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2160261,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfpPaZCRA9TVsSAnZWagAAho8P/RSCaEKdaLIHaVh0jqY4\naa0lQtj9a7DmPI4bqQerh9QuLvkkPCNhRETejOlSK9FfReSAPFrTqf6GqZnR\nS+Y/6pzB5cuyxpcnyPovbmhMn3ZqfH617fq9G94YNyc+39rmCgpnAd+o/cfe\nQJwzg5/htIql71o2Ml8lFitbj+I7pWZ1QVgv09i5eb7HczGuXtxy8SyaHgd8\n37D+l8wCHJmzo+PIMthW4pPVEaV/IhM2XdrXpxnnx+XpCMkRqTjXU7zf4nPe\nKLLOvqEeMVDYkjQkU6iCooq3N45nkPkHSldstbGJQCB25KBtIyiUeu3/2kQz\nWvRFAkTeFYvk/Hg1/7hmLIveXfPdRySMlqhUZhCyXcwfeGt+d+cYQcHoMN/E\nlgWAbFagxDuRfSP9L8mn0rMMmSbf5XyKkT8W0NUxeXS9N0Lu+TCg8RCe4VT2\n/EYhTT1I53x/MsxykYmOB5VDx12emgAFQF80WR5Ue8y2SPwm05X1b7T8+nHb\nS8k/qLf57CutjxBOg56T0t7IdDrXd0PEDiUP9U5kxglBhlRJ58aJMMnS24Vu\n68V2PAjZz5Dh3d9iPNyD3DrsCc+ZXGniHvnXJQiNJWt4tUWl4Ot5KhLoNZ1U\nGSMj5sndWZNj7aIg1LDdckmjq2QIdkxXIz/JcIvsXHPWrUYRQx6gubMnK/IA\n7Xw8\r\n=Gcep\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201106.87dcae0.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201106.87dcae0_1604646552428_0.5730459072733463","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201107.23b4af1":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201107.23b4af1","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201107.23b4af1","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"7f8e5fbf5937a12a124e265b05eb6142c3fe332a","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201107.23b4af1.tgz","fileCount":426,"integrity":"sha512-ZxpADlmEcx7XqjZBJvKix6manUNeW5ZnMFnvd/k7QGUQDxixxvYpzX/P+jfkPN0JCqEWqVky2Hkr39+MZxnYow==","signatures":[{"sig":"MEUCIQC6dAuIvNQ+BXdHTPwYg3Ehso5tJ84JogKNFaSx2KsmQgIgLDxRTKP4kphMW28wYhFvR91RRiXAhhLDfNSI/WEltr4=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2160261,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfpkfoCRA9TVsSAnZWagAA0wIP/2Ry6SuLCgRY2fNaircU\nEpJleyTLhSvuaFwba6c0iMau/2c6tTJSBDnZtgtPRkwm3R9i8Hf2foxyF5XO\njGhW5NaM33dgp8vbCTFTilCw6q3wnYMO0TbkpdaYM3iN4yMP9gIT5Ulto2HB\nAcuaAYj3lVNY6pki1LrVGgjW03KM84CQUZyTphukQRWYXQ+QClk3P5si9qwB\nvH6GtiVVBBC+kbs0ytlLsFkE5676i8FOZXyFfj+LHBEgedQ9LbSxhZ5TMNL/\nsZQ9iaRDfQheQRTlcKD4uDGod9Cd3a4qN3Ux/DVypOwYMaXQtAyMZwHg54UP\n1dBfsAMbBRpqC60FFEsRhGx+n2AE3NTMZfjE8sm/z/IT7hT8tQVkbGI7czwE\njhUk62oQEXjoKtAIwNYQZNdeKorv50oSnaQY5xiWDpBjbUxFCow/6xfntHog\nlUer7S+rCHQg1lvzOBuw+5x8Ax9DO3za928AtwKJFsbWELfugmi75aTe7STV\nN6QzeNDNoRtkIdi5ZY/oXvw0n2UhsgpYDXVnANxR3irw0M/zpMUrTZSbaieI\nxUtgBsk9jyWz4vjm9byVdrD0hSHByc0mBz6hCpFTnqpCcB93pamRl2X2Po+3\n2DHwU5y1kmkFUGZFWIslbgpBUOACc+KsaYNKuiIbf1x/shDi8TC99VqKyB7T\n4QWn\r\n=ChdM\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201107.23b4af1.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201107.23b4af1_1604732903410_0.3463022539689742","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201111.10062c1":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201111.10062c1","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201111.10062c1","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"396219d5fe0b0b85c9697b0466225155920a4a1e","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201111.10062c1.tgz","fileCount":426,"integrity":"sha512-Tsga1Ohz5PJJ42xqowrr/6IPfPUDjJHL4QpM57/JJTV+L+bWJV64V/NIWdrYJGc9ym6TV9qSFTFoXTi7fsOg8Q==","signatures":[{"sig":"MEYCIQDTcWbbkKh3AiEaXuJ36dvPtLO1N4xbDLYYxDLRxwQv/QIhAIQpOhlt2230MZI7FEVbTbla401biVpl/5BY1eXqiU+3","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2160261,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfq44rCRA9TVsSAnZWagAAcvUQAKEB8Is/+lxAs45pwXOu\nGAEVBSu6WRgqCUNSo+7xWGDa5dx3t3AkpPjqyvv81qs4VquPOlmXsWzsz66K\nWxh8GSMgaLW5tSNEKn2UG1eDYjC6JEmL9mXHTv6eqxu1z+QiHGzbVuxKPjuq\nP9C8DwPxRLrwyfVCvNjHaLE0Y+jJRqNHUSbJAo7WMwkglSXxMUWVi/B2ci+r\nYjvhHVLJE+7i4aHFCXYeD/hMPKX6XOLJMMd2I/J6J5VUPk+zP+tivkqqvNzf\nNs+/vsBo2fMdUdXBHEO4KsRs7M6yxc07/CGs9HEaXEaJ2JBVBGefuDSUfZNH\nd0Dyf+lrnd3DXgA+yR1iNJ7egDhuTQ4ETgDUqR9xUJGRagi+ZFay81CYmnzX\nuFBKRPUPIu6QrjtxF5ZtQ89D5h+BFI4NNret+UTEOwl3umN40WoGoJF5lcqR\nAgD1v2dGCjKeUBVvAsQVb+HIRC1qkqvdNCSSTkP4JujeKFKnORE0M8UN8MUQ\nwXmtkO2b+R4br2tESEjLEN+g2d2zLTquaUShX8xOhkgKnCLPVgP3wGIYYgcQ\nOM9RjvwmVQ4MKZl6oZGCW+z4pqlR6ZaDEFLujLW0cyDBDU//cLOrU5yicid+\nCcdfOdCj778AzmrwCd7n8rZD1l9RMCCPAy4etZezUxMwj0/IRA2p3b4+dvIg\nR8Ri\r\n=LazY\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201111.10062c1.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201111.10062c1_1605078570783_0.6882876296553258","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201113.e85b53d":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201113.e85b53d","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201113.e85b53d","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"a674d04f7936f66c0629200990c0f3bdf8f8c73b","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201113.e85b53d.tgz","fileCount":426,"integrity":"sha512-S1xriVwlHvoyAb20u3lo16M3mq69blmfwD7yR3tGNjSrz8Ls7GQgLuYDSFmKD2wLH+gM1zk4w1eYLGDs0XG76Q==","signatures":[{"sig":"MEUCICTqYCpASkWpxJGwvQwr+OdGmaC4auh1iAUKkbpjDCcdAiEAyWhY85WgWZu2/0xVa7cHDMQlId9UbTeuv5G8w6bdAPw=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2160261,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfrjEuCRA9TVsSAnZWagAATO4QAIa/l6qghKba0U/6GqYt\n+oq+QBbK8fTpSXUWKlPaNSvuvBYKjCGFR94qOVswTstLy47/48UTaDf8DaEX\nyXjh8x/N6ChQD9I+wV5ltQAzteVI7L7vpBIUn96UE9eUSDSEJguPcHu0f65U\nNCclD1LmCXZ9SFXmsXEULvVpxnPg++ASqISJmsYASQBBl2O3N5weUjd9m94c\nZIDOpx5iYqjWiwLfZSC2ucjTlqkgC+Be3jwIfLo87iZAdIL1azImtFHirbOp\nem5W5aNqcCxYJGRYiCziFE8ng5LAfz4xT6DvZQF1Yi0YigAzgI4F2c3jBJlu\nWtjd/1jgOREn1IgOU1PiCTyqiBbYJTVQEZRkXkxmSMz2aGnadeLfxKSpd5p9\nmpp1YlfgqUuQJRzhxCiV/Vp9tB/2VN0qkuaX0TLzXwg2hmxIdfk53FrJkXG3\nIfOKH5bsRei8Tc85aCDbf/HEGoNpu1HmMZUKnEmbRzaeoqK0knEP58npqs7U\nXCHHQ+qmYnfADqPF64mqQq/YRB0Sy/2VLMuYENAvfmlWJbOvg7sH+GPEhKkG\nAH9IVc6HzHcl03xTE+dWCz//g4YyxFsiKx7KHIhj952CtrGrQ3aDC2e4umN6\n97g0oCrO9rJOlLX0sEYO6KAAHpa9BRUXvS2b6VmMiEjTrafhWQ3vH/lJdF2t\nBiKm\r\n=2THq\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201113.e85b53d.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201113.e85b53d_1605251373877_0.5402243512363232","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201114.d501bf0":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201114.d501bf0","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201114.d501bf0","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"3e6105ab82e87d69ca839547570f1675c2d04025","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201114.d501bf0.tgz","fileCount":426,"integrity":"sha512-0i46qHV5A7y/GudFMk3qtjqLwgAeV4Dutz+4YbzOcRDnFEur/3AG4091dTe0vUKcYcNhXc92ykL0T/yt1QEyqg==","signatures":[{"sig":"MEQCIESmt+An927jHvB+LApOwkyJCAN4Z+ArMOwEaPOaRg/hAiAM7BN5BmpDstBAYBWKjhQ9gyI1xAS3oBYfDW0/gnah6w==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2160261,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfr4KWCRA9TVsSAnZWagAANAMP/2e5PJD6hVoM9jkWtTn0\nUWKsfj+uKkg890tqGquojcVWq/SN8VGraeONYqz/nY6J10u51AS0TUDT1vlE\nmbkxK+hhD84JHZSv5fGgNOGJCZmVMMMPvapXz3fGAQPvyJuvtus/LaXBiYQJ\n4jhHE3UxyYAQp3+xiSiWPCuMELn2PQ+fEPXmkLKBvGiuFuwK0YQqqoSIcWck\nu4F+RaVGW6cXfxMH2twb8AFTr8ApOwAnmfZ+AV1XMfdcBuP4oEKSu7bBxIls\nX1sc7K5IdHOP8YYIDtm9bKGx5AmrSpIqv+btw2w8PVclP57+zVTLIEhAbPkg\n6/81pd/0DCHqIVmAm4tcLEQ+LCF4fOoNJaD1eosTVNTlLs2FrvOZDVz8ZX/Y\nw9spnAZVKJIG57I8TmJcmpgnTOr5duFtg3qRwOmoRGveifnv/y0i2U1LyPGK\nG+3C79Af+b9FL2dQz8HYzOlWwJjKnixMI+9M7c0X6D3TLn0tfZP3n2UcPW0H\nhTiEC1kZ+ZqCtwxqcxPL0ly+Y5NKcCc7j1d9IIj++PTmAOLNdq0dphsCz1mJ\nnB1DUJMnxDUZZ/46hGZmKo8fgalG1rG4xXf6zsiDHMCw9SBgR+RKs72HY0SY\nG7+ldHjS7q3jVDm6Ifl48WyS4IYfXsVSIxeCmHdRsqxOLfCs90oihjCALu2w\nWFLe\r\n=ybGe\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201114.d501bf0.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201114.d501bf0_1605337749626_0.04963985350282729","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201203.9b18c73":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201203.9b18c73","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201203.9b18c73","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"f133f1997a685467ddf88c5a095cb98d3942461b","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201203.9b18c73.tgz","fileCount":435,"integrity":"sha512-L/X9K/M0rk4RWE7cuQDgocP2+XZ4rFc4a/MzbWLkuT//6YvXDl/Bl7fF/Tpc8C2BBLqnXQKepJKnF7nwoLdB8A==","signatures":[{"sig":"MEYCIQCw0UFU2+pemRGM5CFxdAsRkCxdCp+XtIOQucq4C8/V6QIhANa7r6hzukaCF7BimGh83bPhJM89facdZwEURCrfxWOY","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2173557,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfyI8RCRA9TVsSAnZWagAAQ3QP/jHYaLRhzhl5XbmvUXOD\navND0G5DST2xQgOH0Z89bc33ICQxUGBqix3rfaaDSsYvyNmZk4Wp1rZtDgUP\nvFSI5BrHLQwIrqjxAQlO7/e4BUO5kHKeQoquUfIWT3czCFWmVRGXaqU+kKoa\nclXX5t/21poDrW51Vr2wplEuZD++I4NtwrrngR0XGPxgYUCMXBOJBKPOGTkL\nZG6BG8uLSV7/uNLmfdbxI5H3KXnN70M3J3sS0lcNsvifMXzqZAbct6rc2Q4L\nKkLttICFVnfVpHVfLUZ7GQdj2qzMcdvzL2EFWvqeGG47pY3Tk1ozeuDvfSpZ\nrriTOUWHyxpTh3rKUnrCcn1MU2WaarDc92BdW3vdexnEX74ek90pa+stqL2q\nuUuc7r9ygUQrnuXqX6nJ4U9/3oKQJVfncgrhK75d3kP0GyMjZILNIcE7hmBH\nMHXwFjzty9vnFqEU6ZDchjOm8GkR56k8f5YLc/Mg12MAyRrxBmQaVn32pWg7\nqfYyE+YI0oPE56xpIX/mkTOEkjjBn4Wo4bfnDCv9Xe1nsYgXQFKQNOuCAh7X\nlfRDxw9IQC/mus785m/Y60OyuxvBqrzhEc4ER9HI5lkDukHCLIIYbDiMCTv7\niB/zIMouvZ9tVX4qh8GrlawWrLqPSgMgNPE4D4kJKERu6BC7EJ2lETKnUWEj\nYgiq\r\n=+Q73\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201203.9b18c73.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201203.9b18c73_1606979344631_0.5337878613694473","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201203.69f8eac":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201203.69f8eac","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201203.69f8eac","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"5b6a93d9937fc56c95b26c78133a8e970f1c1499","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201203.69f8eac.tgz","fileCount":435,"integrity":"sha512-sGnwntDkhbBaxlYEJcBvF/mXA31WrVgBkUyIR+hiMstCe0Y1nF5YJhaMz7cDEOjxQpnaPwIq6aCEx+1GE0aDXg==","signatures":[{"sig":"MEUCIDEGtyLwFVwryUyeTQnGh4d5WfmIZGNmc9yCRjngT/IJAiEA3N2i2AdbpDYpBtvazJ7XMvPVfA6RdDd+utsR4ENkvew=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2173557,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfyS0RCRA9TVsSAnZWagAABnwP/2wjrKf5s34sEg8r/MGW\n9vyHNx4vpzf2Fyc93gMtVGKzhrJntUWrT3fNW3omlyhcL1Rvb2N+KV072m3s\nxhmK8GZpMz4Bdg9SAGvAcxyaTT3J6wtJu9icqUujLaX2NrbG26LBDsIrrOQv\nY/BLmdUEwsZcLsgHIRGJQToR6prCWIdq77liFm3PMlV7c5zSud76nar8V5Ps\n6yuDDq8Dqau+A28QoE5lDuoWxKn8EPiIvBHP7NPTnd7rwo7bn5UOnvB10Knm\nNWBGxTrVtiIYU7fTEPVTsuieTtldiHn3z95FWxvmwZmJ/6jIjhh9Deo88Ubg\n+pTzwJvRLyYPo1lDjRWm7RloI0skqMglp4z8PJFuG3RYpQAvBOrDysij4GuM\no0T4nX19vGHK8zuGBxTIwXEkVHiGKu7ztxwAamHQz5Y7LGaGhkXo3eVZTUMo\nF9EObMeg18aiPcYk3iaobRwsiaV8qgEdBbE8Krpl+Mn9+tabfyt7CqTLqBTB\nWtF9IrYhxCiL+35ksWzblrug2DnOsPcSwrb2/T8Hr1g7ls51uxp0yRUa2WBS\nyC2rdRrBZKCa2YQCK0VP2eSM2JpthNoPYw6KygFNzqpcqIQnKB9dsoNY5laj\ncilGjHXb8vmx/z0jZhXePA3/oNJOcoEhJfQ6neifSe1MIZ0JQchCLg/sj+rd\nOTUU\r\n=s4Td\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201203.69f8eac.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201203.69f8eac_1607019792786_0.7256244664286979","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201203.cd29e25":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201203.cd29e25","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201203.cd29e25","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"31c86be4edc44a8fdc044fe217b31d2d3d2ab0f2","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201203.cd29e25.tgz","fileCount":435,"integrity":"sha512-wLUdBXJiwMjLcE3mSYiKdpHOUOb8/LBW1ifSVv0m0X3uljIO0Lk6d72qwh2sDIOkIzYr8iLIK135xtxIxKumxQ==","signatures":[{"sig":"MEUCIFG7qxfQYvtdO0OEwjYKO8tZfi5/w3IKe6ssEQq2mgdpAiEAtnmax7bBVJFSWh92VKOQNB8J5cEU6FycA2uC0u5l1Wc=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2173557,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfyXPwCRA9TVsSAnZWagAAnlwP/0lFPhExD5orqq40ZR1L\nCCHtxFsSQmmYPp+2sqfz7EV3hgfH0l9QQPSGPjVDClkpcguJ8Hd6Cxj+aOB+\nEDQeSyk+UZXcp0dmyfbj0ZRMUgDRHLaQxHmkio4jKt/0YBQvZZVNOnLu8za3\nw6sYKBV1sDIrWvklIV9Yd6mpWLNRCmSoiLZyLGDVJZnEAV7vx9KRvGUBagMX\noX3YPqtVrm5vDm9dusc5dnkS2pBfNkNw60SwcjoPQLrmajql0/71exwYog/1\nsweJJE3pcGtSH5IqqlIKxQ1yyiNNainyb/t4wIMJyAGNFVk2w1rNXrHyEZ6z\nuQAkA5SbLY69biw8ol2kNi2BU18rm+vCrtdXb6kKxXG/oSVQ1fQSBFSxZgXa\nsJCM/f/iDgm1fUG7VEDA6R+OkYs7RoBMR3oTLo7sC0tYyOdI6sn2GatOyDwF\n7B8LF5L651p3DDdVSP5FN8Jse8XY4GhOe6d9NgIo4d4gx69++tIgUjWW0lRc\nf7B200/B1Gy5Qd9DnL1Q/9FwcfAdxqsft+B+GnS3MVDK+Yon+0Z4okoR9+QJ\n1pjAaD4fcwkWjHNzZs4TTAyVr86ieF2IkNFZJLbwQE9xwxRRYS4TSVxnoaD1\nc6RVEFbNP3OiRNxRNeG7YXawzWh5UXfvthFkuUrBJQLhlOp5ZjCOjpUfEVOM\nOQw4\r\n=g5UL\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201203.cd29e25.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201203.cd29e25_1607037935519_0.7220938650991453","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201204.cd29e25":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201204.cd29e25","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201204.cd29e25","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"45c9d99af580ad1b4e5a3fe3f556c3d1a549326c","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201204.cd29e25.tgz","fileCount":435,"integrity":"sha512-woWrnd+OuQ6k1dPiJt07Eim8rAXqzd54IG2wkTNqeU7aDbgaZ5tYFLV98vlGvw94uT6QWOJ1w8+JWeWO8aDaYg==","signatures":[{"sig":"MEYCIQCbSF3VtbEoEIOVVo9aRJwhAcaJ2aXdaa+f34AzlHVfcAIhAPvrM4z6BXnfYhOTyCd81iHg60advPVxBbtMo9ZrALpA","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2173557,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfyeCpCRA9TVsSAnZWagAAMAcP/ikaK1wrFl73Zf/OK4nH\n7le54dg7w5zARGjI42jHAtCzQeuA7k/41mzWSG+LB9V1zyBJ+U4UR5vXQ795\nOiZAGs04iNJ8NhZECGqmcbHy17hy+JuipqOoHzBFGClmxdFoiGaJc7A5KlHz\nRqqpLYQB3MmYCcr4KkO/rcw8hftkrHRFA+dGb/SFGT7jCf0H+5VuwPZyAMJx\nwJ8J+f/TDi8mRS4aX097Yh67mwgdQKU2eLX7zpXjzEhkAb+4vKPECk71gP7m\ncUI08hHezDuH8QWg4Wcq26ydsVmdaEBsT6HVDJGv3k+Y2L0rPW+4i6drjvSW\nd31ATMqpu33C7ElKzDasyAG/fTuhds5c1h8HuWKh93L6v+f+IEFFId/zBRQa\nMILbDzavwv4uUGyqek+0nM6bDG9G3JyX63x4pHIkcK9Er67oIYkVgascZWZX\n5rHaEEEyG5w/MSnPJDkJDz08cCyxUdz6mcXrxdLfKUpSMhEFEFOtAOAiwlvb\nrcEc8B5VfpM2WpzyfzHJTAzNxSGjf0U5UiozTscIqTZqBTBWES9NdB+4IAuy\nP+l68+7g/KNi5cYHq0XTp1cXkRq2JgIndSTnezsGtH7J2nnHTQ9Vwz1CeiGB\nxP+P4t7SYbkLEqG1K2yeXL8sSon8oxBc/cdd5NeCP/BcOXTzTcLzGx0Brnlm\nnPXO\r\n=k9qf\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201204.cd29e25.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201204.cd29e25_1607065768400_0.6077341942461882","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201208.165a6d6":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201208.165a6d6","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201208.165a6d6","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"c13faf6c250ab5b3a492dee80f6712d28b93021c","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201208.165a6d6.tgz","fileCount":435,"integrity":"sha512-C2t9ZahX3Y16AOHTiAFWNwEqzpr60fOZVfZS5YhLvHF8vpQDgPLfUDUrGV0qgsg/V52tUCa/o8FgYVFaoUyaLQ==","signatures":[{"sig":"MEUCIAftSxZvlQBmJUq+8HLzEehoeZ7qNK2x+yzls9k92Kw7AiEA9O8TkX/WBIUGasirZug7CVkBwyAV3lzvT9AGQ0wCi1I=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2173557,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfz9MPCRA9TVsSAnZWagAAiSgP/iCa8M+JCYfor5oE5O4o\nzwQ8akFWq84RnAgMh37d9t4eGhgVzSLPvavmBb9cVKx4uwwlGKkJQp1bJ5HZ\nR/Jkta2PMB66YSINnYR7NefgHbL3t7E+XUuRcZDhwoq7ggRfJHdozS0vyApF\n8hp22IoEmZ5B4H1BTSq8jYs9HmoVwCinbk3tCecM69926ImiTE6TlReodHD5\n5aGl0gT4R8zaEgZDgDJSRNPGQJ8/KWmdXjrp6cw22s10rYXtxUpiJ0r5YTF8\nzPm4grdr3wmaL14TYVaap1Tu9vxed+h1WK843jsODFpWbejXJvZZmANwZCDn\nrNzaUjvsdy4g4gIWQaRppyU5VuOskuUDuP+Ticsjcz2rU7Y0lMZS5X4LqVWb\nd1/nHBD7sLfnbG36YHnquydhvLmFmHFBQeydJLx5YRjoSCBecw5Y3eYpQyeT\nE7rPFg7KuKh/K6WXP9pT8I9vf7JJ5MQ769tQwkL9irGO/Zh+tkMpCIGBiWan\n9C6c8kTNN555h94v6YjcCU7aHcgb04mCVI5+jpngkbOFvAAy6fR2tP6RBLBj\nUjT2C3FwXx8Zb8HVTN8WwR3p9l5pXw/Rd4mr51+Kq2cgV/Q8ifB9XKy01quu\n18BqiZL3vBYfmh1IfJRYhAU385afmiRJetQZ5BevgKonm99qA+9jplrp+OC+\nhf24\r\n=hHJ9\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201208.165a6d6.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201208.165a6d6_1607455501740_0.24983446538760012","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201209.165a6d6":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201209.165a6d6","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201209.165a6d6","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"e531f014b3469487e7bee96071ed36ce2a3aafae","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201209.165a6d6.tgz","fileCount":435,"integrity":"sha512-VPYsKEemVDH7QaAb3nXERZcFZ/usecv6o7XkKI0wZJOpJARNw+NBCU/INRvgAlOLi3wgeNbdagtkRtckjudNjA==","signatures":[{"sig":"MEUCIGtMTu73KIFdBHDN3J0a4wrZvi+u0JPmH0L0a63YgnT8AiEAuv23yhj7yqX6Pl1b4tiSUXZaKSbPOSlVZu9ddq/+fq0=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2173557,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJf0Hf7CRA9TVsSAnZWagAAjfoP/jh1huXp4Er6E74rtMCE\nBHrZ+hpfxGP2YrgWRTxUGX3hzAPU/iKDNlgP1tqQYALWr+rOFm+GDRtsi6Ov\nTjq8A6kTJAHZ0ajWXK4qvWVkIKh00D6JwkMcghIL+EV2gGIKxYAZgHmw4Ewh\nRbEXrCtzCbzT6W6n8f9doEs1JnrvTumLjROwjXvUFg2Yh6ZmNHWEO9lu+VHU\n1twIKwQNk/qApTTFvMOOZPge1QJtQu+ERLOO7II/MEgoGgZagphc17YZI9jb\nm+c1JVlihQ5owkR6IiJRb49Z6gjFDoWnP49Fz2IOuPEMiw+NgC8X41QPr42S\n8/GkhsVfV0ie13vhuq4Clbosn+qTvC2/UEU4scA4ENXYp8UNugoI8FiE3uan\ncWH+L8gbT5T+M5wSafdELhUA1lSGOFqmsyo5OsDgccvuRGzCheAmip4Lk5Rd\naSfEF1iYgN75cQqFe70Ux8ZuFeGKLtWpdL/RIKKH9jkV0KxRNmpBlZ4RReHS\neb8X3QMIEnPWRMc2axV/gEOlHE57q7fyIzRxpwd16S6rFT3Dz/qvEkWssCCR\nWOY5zAIkOFeihdpyp5BZYXlPa8yYhl+7/C94I1cKrtfMSAsoeCnaLsEf6O11\nHb9ntBqca4wEeJoUojxMtC0mJxpopxz3cV01NUGsVbOSBk6jrr6JzVe3zG8i\nPKWJ\r\n=DByq\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201209.165a6d6.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201209.165a6d6_1607497722354_0.5956613683309491","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201209.54c74f9":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201209.54c74f9","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201209.54c74f9","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"8547f3ad0e6ebefbb9655367a82967f1c8d1550c","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201209.54c74f9.tgz","fileCount":435,"integrity":"sha512-tUgMPIoE8kSnKxgQ1n28z2H1d9eJXuCojHxuAalIJRxwNpJlJ7sOL5g6c0pTwykNiEzRxFkDkELomPMeiJERSA==","signatures":[{"sig":"MEUCIQC7F7fN09oCsfUwIQNOj2p9USZfb7mCTS1q7A5kkdXCSgIgJrz7xdigJBzLyUlNPX2GTsRbYJEhxoRR+rbGVrIvPMk=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2173557,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJf0TuZCRA9TVsSAnZWagAAjjsP/3rS2ZU7gj8CJw8gY8eO\nSuI+Y334c87uyxA48OpIPtTpiOWp7B59Mr/hlwdXbnk9AcUgvQffirx9+LBQ\nmD0L1WywtyTHAohyT9GiID6SHuK6qcvMRK8PPA6PumU+KoSevVq4sesv6Cyp\nYg+i0/wure6Od5bqldkvIFUmDlhbO9EzauL6lMXAMxPNv3rGUP7whBDCMDYE\nsCL+5xgj8weXHczzMXunqzuRZnPzpvmSoJwYr/yxxrTcNzeHffem0xZWSEGu\n+zOrrR856M/HL3f4+Vyb9I1ECt8wKhR2F1trHvh5DvnhceMvYmH8hqHsHQCO\nEIXA9wF6WX1LYJbRaYGoA4iWlkZzvjW473Dco08cMdRC4aM3AiwC99iv42t3\nMCcfC2T81kFn2fy4vyWKr4SsdKWgOr0F2Gb7kENiQTbFc4IFWP1x/BrvN07c\nFKvFFe/ns7rCbDw8lJcxBESN+eB8F7LwXeENicgj78hFI3D+BeXDqPnjj+46\nKTpQ+48QmgJ+Drp3VvvfeAZ9ad0KrI8oE8/qiKfQeWpxs8LU93bdPGUyiXRt\nSWPnp8i7nJ/U84xBb66CCyXO2Tyc0I1/PoO4WOHlgv+1FtwjKGSfwJFu2Eic\nF0/GMw0Bzr1Uyh208ulYZ++Hr+EWkqNyKNO5G8WFCX8wDI7x9hDs+H1oXQ1X\nuYKY\r\n=pNpT\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201209.54c74f9.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201209.54c74f9_1607547800531_0.370515506192171","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201210.9534f1b":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201210.9534f1b","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201210.9534f1b","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"51919aded13aa9f5e29c7af56918572d4afaca0f","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201210.9534f1b.tgz","fileCount":435,"integrity":"sha512-lVA+3eRgkLdS7X0YkJB0At/o3k2ToaqllIiwDtcMZF/fY/CsekBFajdUB3ZbfP4uqETiDL+JnItCzyK26aF5mg==","signatures":[{"sig":"MEUCIGJKY6I0xmNYWp+IpOriM433t3ZSPaW/lf6AZQE/to/EAiEAlUw0pOyW4wsUSk9RuKORSgnQLpAgLsk2dJgCX9aO7og=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2173557,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJf0cm1CRA9TVsSAnZWagAAx5UP/j4HPaGk7FHpy1hZvsG5\nRBga0S4hEVNdOvkmP47IYYP1As2K05sWy4swH/lEpqltq0MzfhSlUNuWNb92\nb13Iv+yc652n50nhYcECutlvDvc2MCNCkL1Lirm6uBva9wBWlbi5r+S+LlLT\ngbPyo1bAvK38Iql4m3cwlXJUisjmoV31eM8uFJudBYAh9I7My/Rd41S8imZz\njjjL9HwU88uKJ33baWgpzEp8ZuHloOFtWmEtqIKIWQxKo45mEswxAmUOBKYW\nwydMIMHdMQBDT9wRXDiInCe8xSdzyKJ6dJP6jEQFIxs7IenjWI20LQFHRwSe\n8jJ44xo9uGSnsMYl4q1Wsg9W47BSXitU2feyVWrQGyQP5tk/9NhxBZUK70tl\n3fJTV/nVEPaqd/k5YNK1vQoZ4gSxbVWtxY4cETKl2F9cqddDsmjilEWIO9XX\n59LoP7DrQINm1Tm0Pjt+pxp5JzScI1hQgvKxoYbh/m4KVpdo5kDb90UwisKN\nbRr1euRr0nTPAdGcODqv4pr0AiLDPzZO98EUUsO05jcusiybTJIzOeNy0F7Q\nOlVCH8iqUBobvzlSxxlDLcWvym8jtPy6DNU9CBtfzN6TMRbj0u9hfEz6YkuQ\nF7ZV6pVLmZ/0x+OLyy+UYKcPag0wR8/14R9+m3U8/5FlmjTmQx1Tler2Ahxo\n3hnt\r\n=DGhq\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201210.9534f1b.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201210.9534f1b_1607584181342_0.4861260946336923","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201211.b852646":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201211.b852646","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201211.b852646","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"47c9807dec1a5352fd9b03a5a4b995d9228c9d5d","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201211.b852646.tgz","fileCount":435,"integrity":"sha512-yZ4rg+hvlQbz9IT1PadpS7I2JDSdSJOAGkrC6+NYoKeEFWvjBocBJxAjWazZRlw+3ufaCl+lPGYMbjKLY+M2Kg==","signatures":[{"sig":"MEYCIQChwxFK1ovKo1StWsu2oqqkJZriczbC0ohJniGXctYk2wIhAKRFS6pYx56xiDhaheM792FUHAw2sCQNwKGJFrRuaicF","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2173557,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJf07XICRA9TVsSAnZWagAAVgQQAJtFLDqZrkmZbBNieipz\noPjzJHJMOUkmu+ryTF9FVhJoNnZTGv2bH0fwPkjUOvS1PDt6sbBbn3snaRlB\nmEh3WwlV5MgnKnFLsovZXjnbIkWADMCgzLnVAx1rJAFRVrTGXgEn06z5jfG2\nI+0jSdCb/iYvpcjGy140Aup+JKq06lojB6ESZPe5OVI2SYQed6UapTDBqlV8\n6IwiPdn49I/f1EUP+YPtPUTNey79g0rB66xODU1xdRli/Uv3yMikCvYpOjhU\nO96sjoMoGY0P4xU7O8kR8ImZJf70rZ/j5IUDBu1+k2mIYdSnKHc2reo1BtQC\nG6LKW1iZrcclDDHSQ32O/VmjOivuc6UanIHvTEYubi6RYA6r1QVgBas4LCsW\nW/tlYbv4CZU8e1cZEFI0FW0wzZMCTHksIaJf2NU1xMg64jAS0rNyndCERipO\nTgbXE3/Z52lE97A1TkZGeHkZ+0puMlX8kqpK/SVeiieMBEWSPB2Qe0JX0pR6\ntXSHRWWpLCVGfMw3TJ3tFL9KmERyyYI6sVWNHKW2seRHwrYqBzuJQ2v6TUvK\nMx7AhyYa9hQtu3n1qvuBRuuAWpJk33V9fpRO8jiTpnWLiKy+TXwSbXcfdlRq\nZN+Yt6o13E8pqxbJO6AsjSSck5aS7hFqPloVUFge1MYlRMNGiifejo690Crk\nuzul\r\n=NtRR\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201211.b852646.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201211.b852646_1607710151263_0.9521572527851989","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201212.7d972c3":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20201212.7d972c3","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20201212.7d972c3","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"582c5360c0b32cacf7534ab815bb56bb56b6f82d","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20201212.7d972c3.tgz","fileCount":435,"integrity":"sha512-bp7ptGmi4sQgDnFBcaWnqmzMcmWfo2LqLZ4IWgBHlxH3aI1PZV3VqbebQICWqLizNhPbmmvEICEdy7Na9P+ByQ==","signatures":[{"sig":"MEQCIB8MUboJD8N2e9AAAwd6Lw4lec+wFUa/IZjacrTqPlpDAiASQHvGYgbicO2yx3nzP4i9JndY9eOlh7+Ft4foAA0TPw==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2173557,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJf1GyWCRA9TVsSAnZWagAAdkkP/3YiPIeR+D0tlfGnlyj7\nONEXAbAdFX27wM3KmYZCkawOdcQticYE/MOl+k5G18jb3Bx7DV2r5D9bugSS\nFlD9humkpm6vRIkZ0fXEvzOmKpnvVhfWrn7T3686Ajl6hD7vBmY74lyLnwUf\nBOMhuW6lLujM1OA2CmqlyDz8codowVHf5l1KLkRrCV7tYAeG7fSlN2DSZJ+i\noC77LgriRoJukL7DsLrf6KxBaTUvEFVurdS5hQgu/a3MDyp/8DjR0eQqk3S6\nKWk+EpS1OZyQtiedj1gxrorSie4vVhY7nNv4HWLWzmJpu/W4vwdFTcTwxsqm\ni4UiSBp46Wl9UaksVRXCGcrdXNIzelQBuEmuZBZf6FeYiK2H6bFg805TYL89\nnXdcVvM+GY9hPEu8eadFzaSQrzm/MZ+/XXxLhvxx+w3epSvPwxYvHD5gO+DN\nonUH6ewBxm1Ty+RloMZ8b2ATfTlyxYzAkXO9r5SU/JPSK19VmsxEmwFCCwBZ\njgzKorrrXkTfJrpHGfBjFw0QLGcXE7Gj3yxkuyO4v0V8B3NCdwgWolV5L62p\nT6QlTSMOeAJQgBQdZMJsnCgaYnBCFeLPfDQnfbOt9+SQbwCM0SvGOfMpRfJa\nkYHKndkikGI6TsNzbTzTAJj4ik0JOOLiotkQt7F2vmH1U8tH4BKY6SeOxpfQ\nssBA\r\n=VqWB\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20201212.7d972c3.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20201212.7d972c3_1607756949323_0.6656558465481739","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210124.468c75a":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20210124.468c75a","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20210124.468c75a","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"97155c973aa0729d9c867018759ae768b254e321","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20210124.468c75a.tgz","fileCount":435,"integrity":"sha512-6gsKNX+r7cL08Qm83bsGpt+iejerbFriadthkU/OzBcGi4xJ0vnjim/5AIuAI4qWGOOsHC5shxZMQHVv+ZvAPA==","signatures":[{"sig":"MEUCIQDKjQCnlYNXJCT3JahFj8QpJIJr3l09P/FVPj6m6oWjegIgU3sDIN9lL9mR3PGjaHu+7RD2ADwR54977GeCxJaCz3U=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2173557,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgDRzzCRA9TVsSAnZWagAAegEQAJpwVpqHC05RxKeUQxfi\nym4cY7C1GSGJjuv5X7nD29pl4HcxbLmffv1xgCWPOuBsndeBUHTJcr//PhBg\nZpGyL72ejM/rJk0l64MGOykTOq2LeqR1GQtunOVdBZDW+Q8ktxruoReMn50A\nUWOI+TyoXJbvXU2PK/jm69pWFhqmTrB64rLlgBsgG5kFs4dJ6YpmVa1c4PdJ\nbWLp0toIwqlWpcRm6bM0XJJRrnTGZsW6eRcF9PPqKgW1IxnjoEnQxwmVMFmX\napf99dVb0gh6P1WAnOulFR6ZDsQTx2VnCB1fL1KKtDWYLWQyEtn9DcLIQkuR\nrtyPyWstaJu6D+Y37jrqO1+tlImFlDEslnIVyB0yWklgGzPtkpOtjZuJ5rex\n0TR2SNe+ErQsa2tTnbNTiHL6aU6gCDQUHxteahtudh92mcS3Sib4EcY+utP1\nARSGrEua/5l+SOMEX87Hlx692P7RSpPdeLGO4XV9XVhkKYY7WnRLgx2kPp8c\nkYAvZVif3UENox1CgBCZii2sbGoXD1pk8SdZY8YxIm6JnLyQ2Z3/6aapfh7L\nn2JshdL2zfU/Ce3lAE32naX2LOb0uY+Ao2D2kNUamiEXnbDNNixKO8ooWMxe\n800xMGHHj10wWDgc2mn5Wc6armgR0k52sxLkPsOBbRlUbSS+1zORYtuH5/zj\nRr2G\r\n=Zv/5\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20210124.468c75a.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20210124.468c75a_1611472114546_0.4819749131440574","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210127.476c841":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20210127.476c841","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20210127.476c841","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"b7d9cb93c65b7c229448ade5f3d66b7c98aa08e9","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20210127.476c841.tgz","fileCount":435,"integrity":"sha512-fnrpD3iNrYAdxsNClkm1BtSXQcC18Zsz2sMFINmsuaKf0PA1VtaX0FM2VAZlYzxCTV6q+IQiP8E+ycdLdOHy8Q==","signatures":[{"sig":"MEUCIATJnaF6pqFfrQQ/j6ZT2ou+/m5nIP1bbAPa+BSZknvqAiEAtKrxeyZERpR/WLNMozf1gVhB5hz/Hyzf7MdvRtm2yoA=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2173557,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgERGuCRA9TVsSAnZWagAAUc0QAJ3sV9pVSaZGO9N/Z6Bs\njnG13vr5lGblmHm18Uj4QXY3FzwVoxD6ZQXdntARNSrkSLyxXc3l9emBekhW\nnoO0rM61rcOKYzfaIvuyuQpb4qHkX1+sMApWk+dKJXxuU3Ckr5IBopIGtXtN\nckiQIYWgbWgaVS1Q7r1CrIRENAJtxSpDZ/AdU5JJVuIxiq6LYfMLTqcB/wzY\nFPz/zXARD9y5n0IKU/lpWi+xvbJuJ18S/NdB0iVnmMZhQTvF9P4KLy1ghS7V\nioju4HF++4oJBhWF8lE+TopDysad5fMiL1eDV6utHylvS0kHOtW2ngfh/G4F\n/3WDhARP78uHTPkifD7DLvdOa0F1oAP6Xi5neDUrpkAFCQpDQ96bhp0O+k7d\nNeQ8YcireEBfuNDsgYwoNpnDCL4w/d7SIsOiqKv66j6hX5/fP5f5wnk13Vi+\n/jr9/9K1hywdrKafyLVaVOtvs/6esxLVyDbEEYwkadwgr7o1yghEebWwu0T1\n2ls4Kpu+s3+QHAQg8v/1UbXSTjcWdJGfKvBl3/1nJc2Su/xwKAzqBhbhuwa0\nwuyYb/yH7sZqF7PR9sXbVrGrF9saexlSZMUQ3+YMEcI5oEBAnIVmwihYHX3V\n06+ucBmxIKV9NdJ1Md4aKyCsj5mjzQJ/EDrBAxxjIlzhBe7fBGutIeXI4/gc\n2b5X\r\n=FEqZ\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20210127.476c841.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20210127.476c841_1611731373557_0.9762396087970935","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210128.3d35ae9":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20210128.3d35ae9","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20210128.3d35ae9","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"7194e44d24d74aa1392bcf91961f6cc22c60b939","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20210128.3d35ae9.tgz","fileCount":435,"integrity":"sha512-s+2J1ROmVor+unt/JpFBO3yL7hiQTG67LOv5kWbu1+O9z33pgYXMvgXPFRZ/muEU4X/dhRkklEK6bUByhl8+vg==","signatures":[{"sig":"MEYCIQCcfTTFnnjdwRnpgYj2hjAMFvxqoKjLRe7b4eXCATj5HwIhALD+FbMjGjRyapUmJMYXTE9ePAMU0eqVBne3kSx99R+0","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2173557,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgEmMRCRA9TVsSAnZWagAA9BgP/ijx8Y6sy95+Mlng8H5h\nZwBLHamTnvr94+adaV77c/Y9bNscCPWcEIibDoqlN5bgaCZVCc+Ot5GL1rqR\n87BI0oOeLXcn6up3OVyQSQ1MCZUgSSsNHqEdopfeGn2fjaGeVuIu+NtDJMVV\nRHSq6wh5t3tlSi7N8L9JJk6Q3/0ccxML9eSjRt6JJOQyug59IY+VfhttqPmp\nvgvOk1jK9Wl/1AvkMjxwn0j40M7LjWcRtcHUeEqNxjJ1o3l6/3L524+UmFAf\nxECiTnceLZagZdS1JysQNurJHP9s2H+WzVoN2SGWjtK5dUmZQTElNk3sWqJZ\nro5WwO0lPoeXxgGdYOxN1WTbaWwFmFuDUYC/F81BJmyF4f9X2xDVSmRbJMJT\nWxzItV5v7YDnPusBPf3LdznlaWCWifoaddtFb8TJvHuFtXvlLmGnZdAYk1WJ\n11G62MmKe6EdnDGghfxVxn4bSy85vs2cI9UrP6Yq98djSNNvhmZLl5g1x9i9\nHKJROjusGnXcI1/YPcYz0gqnNshuA2d1+vhl4SqaSQDbnW7fLn0tHOyMP8mO\ne8BrrzLBohm9k2wIX9RhlAItB8DAWkHGYSSTh9dgoF8C2PN/5ci01oadNk+S\nNDRFRfPja8RboSj0ZrtoKG+7MoZP9mML6g1rUJ4hIxJs1fr8RTH1WGfpBEmy\nCbfq\r\n=EVlC\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20210128.3d35ae9.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20210128.3d35ae9_1611817737241_0.7582392906713791","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210129.453a0f1":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20210129.453a0f1","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20210129.453a0f1","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"92ede38e81bbd18b2d9c6c1602aa18e702a8eca4","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20210129.453a0f1.tgz","fileCount":435,"integrity":"sha512-DevEFamPOvhZVnuE71LVNZJjN7RPphhjejjF8k/CEmobzBacx5veJx1ZqctrHp4YrODg1tVlrLBLBTJRjBVKlg==","signatures":[{"sig":"MEQCIFNB+hfnqCwr1c9aFRARHrtNXcUZgP6v71K/+1Ru1IKcAiALT8CbqGwG+yidG5x4MG7R6QNMczyKXpybcl3vTgDenQ==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2173557,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgE7SLCRA9TVsSAnZWagAAjDsQAJjG26X/MSGIcaZcjXHJ\nIXRMqT4+uuPnk2NnEiiIIfxWoMBydnxrwUiLKYR3Z4Nift62Tvm72Lk7wNPU\nrI2jT2QZ5bF4+JXr20Iqlao3y+VMZf134PePrVcdrty1tywx7iopUyhmbu2g\nrAwSA4V0Nb7UoC71+a5Yu3K3f4tSBxAqwbTSU+/5HX148TXrFRSKbfEKh2xk\naiCJizQ7ihwEWmcsmaOuv7G32zXKXFVo3H9sVLgm+/XBJ9z542I10VQeR+ge\n0A4y96aathNHPC5GBjZAT0S+4f4RTG1akwC1CD4C9e2xdZjeYDm5FMV8wqmC\neaS+IgL7jo4YcXvEGEjFUF+z4ed7A0IUGPcdorbi12t3IdQkQdysHa2fVTCv\nsCqUQSaRdwi9XT3l47XSSVLvsEeWk85I2Y7mlG4ZtMmtw23ZuxPCEgq1EF93\noUkFwoLDpEu9LpfmVgPLMq2lC/IlSR7diTY6WqOstzwGnc68AyaACPL3Bpni\n1pbC0tuI2TYHER3MkePJ/tlERmoGLfefJyfczpCNq7Fd4gVZyfPgDHjcGskP\n092D05Hwhtm27Lbk+2hYXasW12pHgz1QsfciYD/lgXmWFKX3jaNYkfkt9Pv9\n7wcc5e4vILyDPLtNt7YeSHu/jlLe8rsZ+B+CMkYHtCJd/QuT66LB1ID7jZUo\n7Nxs\r\n=HzJf\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20210129.453a0f1.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20210129.453a0f1_1611904138499_0.010381398883373638","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210130.46cecc7":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20210130.46cecc7","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20210130.46cecc7","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"889a3e4d66fcd1356191bd3b6437c8c21ebf9faa","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20210130.46cecc7.tgz","fileCount":435,"integrity":"sha512-YyUuEeRUba2GwgcoOt3UXwVISkBP10IMDExVUn+jOIk9cPKXB3/51tRSBYiJXw3wjIX6kPmUeMBW1x1783sucg==","signatures":[{"sig":"MEYCIQCX7/tTe0BSC2HwEIYqplhSnzT9UPAh3NV8JlYEbo3nZAIhAKVqTVoihEj0nkXxJLvejc0UybuKB99YeDy7wGN05IZ8","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2173557,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgFQXqCRA9TVsSAnZWagAAkqAP/3xwW95rcHWwkaiX1r+a\nj+SlENnh700z72ID8k34u89aJ/Pm6/z+l2368vTc+XdH3za5sPDaaZltJXh5\nmyxUKodLTCmq6KCboGDNNwUe7H8w8wRZ/lDI9KS5WNyaIL2ZhE5cox3VmvtS\no7BqAHuAdqtC5LhySy6vv3sRlTzn96QS8SnkodJFXMthWn1Zby0VzfOnMwkl\nR7j6rFvIUW9Z0S1pe/hZIztcyDsdxcGvQeubIEXWp1ha3Uqpacp5aX/1p2Tw\ncngZ6NiEaQr1Gykc7KU2g4Z2NwnJfaxD3HZeYGX2A6g8AcXHZHz2IUyxy1Ss\nxMQJGQR7TPRTKK7mhSVocGNoSHB/V9AGobsKrHU5a4nzrxgdnzfmXMFmeeWQ\nfKk6WxiWJEqYZAOGjWt5ozlibwvoY4G/ZYxRdd2jhzgSTISf0SV17XMPLjt7\np4W5BqpF9KoX+RFVAdxtpfRZwiz/GLFartfW9138Gv1SF8v/WeRnJPkXTazr\nDAw+m38RmWO9HcXZWqRfY7wbp6MFGwOV8czGO00vfYKWjnd4bQKgQY7nyTie\nLWgL/vSzZUVnTuJNig48Sd5Q2w3rlzMniPHIx54iwUXHNll/dpEm1cMlktk+\niA9bpqAbPb4v6QrDbZL76oWqScNYZQrhfgXXd5Fy6NshI11HB44RfpQHQ5vC\nP+uC\r\n=LxK8\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20210130.46cecc7.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20210130.46cecc7_1611990506356_0.9493390013767931","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210202.f5bfbad":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20210202.f5bfbad","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20210202.f5bfbad","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"1e4da0da95ab8c0374606864050bfe4c63841318","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20210202.f5bfbad.tgz","fileCount":435,"integrity":"sha512-KrooS1uz3fqPoi/Qp9BmaGjjZmX5yWaXf9TEDFM+U1zhlRUCtnk2Ej58pwz1YRBS0t9ur4zmDlO15tswYzSN+w==","signatures":[{"sig":"MEUCIHX9KEksbmmfudmxpYy4jupLuuOlw1R92d3Q5+EWlh9GAiEA4OwQQ/Aebct9mMUaASq9aCuxUpjXJcakZ+5fDkV73so=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2173557,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgGPqICRA9TVsSAnZWagAAe8oP/RIaQG63owi6WX8++Lub\nSReVeCsVQFWJUS+ZC3vtJOWHweaGsqdzvVK08s86p4pdqBKjph4CjeBy8Hzx\n3eA0yoWcfJgq5YnhvPypLLGjh8WfgYBn9tMOJ6M7omkHyDJHNF7JOZr+HLXz\nqUuNeAMyzZso3yY/bDvHkoKp2I+tHXee9nmbGWqkIMte4/MrHb3u55IBFykf\nwVXaj1x0yO0saT04VTYtd+Dv0tkyFUiRoCZ/O6ijh3ofpIl/MEnOcTfBPzWc\nhwCB5Hk9uQX/82w1/km7P3eiG5Os8HIEiOe95Wc6Z0Q4OylGjrbnqkfFbaWw\nxrunxnLRO6G1X++w7I3EwIWmxLJxCvWnyhgoexmHjjHGmwntYdtFLSmVM4E9\nKAZmaxjpnLT6EgPAKiowVBILI2c5fLLuoiP4gybuTPXaTj4I6hOv6UAJOVsq\nhWo1yVYYFUmUd/ypmHJdY8Swhw0dlco4g7Dqr3b8iWorryJjUIugv51Lut5o\nTY0JENHOKmml+T5aHxI8Wz5Kb8ZJDns8FT4whJmk5r7ChGzmSwuge+5EaCvj\nj3F7I5SHp7haIu70YFJ8wWHl24g283tsExko3g4Q/SCzgniEOiG87zor6ogI\nuEr4HEY7wkK7mZ0/eq3hS1w/O3gt/0jewYkTuF2/PIy+tchb2I4OkzN06/xn\nrgJQ\r\n=tzO7\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20210202.f5bfbad.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20210202.f5bfbad_1612249736261_0.27822758859237395","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210203.b0a8b02":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20210203.b0a8b02","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20210203.b0a8b02","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"6ed310c6d872964d1cb80523da22c5769b039470","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20210203.b0a8b02.tgz","fileCount":435,"integrity":"sha512-0UkMBiR0NDLt2tJnq9q1LUtGAgZCejiJf96AwNVIPyX7RVznP5+JTrN4ZaMINyd36q6yCLqskTRLa6uNI2gJiw==","signatures":[{"sig":"MEQCICNZekH81w5uMurRfjKRBPlJXX9FGgC54+J1pfAjsTVXAiAHHZINxRO8cDIUubIPMxUj0g2AbYYn1rwWG1J6pFBjHw==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2173557,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgGku4CRA9TVsSAnZWagAAdYQP/iTOPzPbU8zudnDSv7Oy\n9GPyqfCqBr1JWuinGbltLvtizYrSbM1g0L+/J0XFpimxqjg8CFCFJU7ewW0e\nWeIGRwFnF3UB7SzAuuKu1oG/8APc9S+ocIAhhSwMmNFVrdY4soPqMHMPvwpv\nsfHjarmvEiShsqrtAR8aB6DFsBkcRhK+duz0bu/hLuAF0ndnnNBwfqqhz8Di\n4p0NaECnakDb3bSySngWhX0P1ShMZcm7AWy1DtE5m3NY0wJ8i345Oz1hZB/r\nGqOQI5LClXKQYMUJ+p6zDvUG7GNtZrzSPUEjUmSVB1VRiYpEyd4bzQmkIaad\nAPBCAwc3Eml1pSviH5boolaZPL1j3tbKrGitiMjY1zcaSElTNCwSa5Epj2NT\nSArre/jyCae/8ZmA7mF4uhl9b2Ip29IZ9mnP+maHvbRieHuLc8KTWjypZryU\nSEAglgEaanUYe39x2ftpyfvXw9m5kUhWppjgJSZ4zAOSj5ai53HwK2h7FLol\nWbb24pWSu760yx3UqYeusC/hrzUs9etw4ibvH3AyvtUqUo/WpYyBfmITW2Sb\naff0ZgRESVEzs3mzVBJzB9ZCIVeReXbPCtBzNYd35c6Pl+3GNg1hHsitpiF8\n4t9A1JfBmhbfrQN+Ztb+fMQfncLV7u80d6LEaUhiNAMEvcyszkfARuenKPZe\npSXl\r\n=a05z\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20210203.b0a8b02.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.8","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20210203.b0a8b02_1612336056131_0.4118674999633949","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210205.12121c8":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20210205.12121c8","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20210205.12121c8","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"bbf4d32b89806069d87e516cb82542599987581e","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20210205.12121c8.tgz","fileCount":474,"integrity":"sha512-37Zywtk6rwbs5UYRQJCraE4PDa/MmVi+zBNoC/N0ma2vkCLRsY7mG18pbbaTyKUaVtOTnipPfaZwefXpWJFEwg==","signatures":[{"sig":"MEQCID2ov/b24JrugGF0rIdTPEdX5pwiJg4v15UZDM4cDT64AiAI4+4aNZ7FIeJySy8kIMzW8u9qW9BeLEJs122nue2tdg==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2376591,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgHO7MCRA9TVsSAnZWagAAFWUP/2tj71RN9ZO/FPTEW1KA\nPZbQOQ7BTmiT/H0H+2CTFT5ut9Ki1ICMjq8WzZUAUmJn8plQZVrmcWn63q2/\n3zzDzjvJeCOFRq7sMqQVdqiNUKDhimwnhk74oPotufV5PBFh9RN5qkffyPNk\nnITg2+ZRtdZIzlgaOSSgDTxJRoBvDtyuyqZFGL56nq87tKY7+CqBejuvyXUF\nnxutiXWnyusoE7f/gArcGA+y2oL+j6HYB2P6KcCyI3PtsfuD50DtYwOrATKP\n9TtcI6N1bM/RfjorZlu839n0fYub7QJY7jC2di1dXkEg5dVDGS3ytFtWeYkc\n/id4skiEdcrS1MK11mkEf2jyObZeAvYFSmTpLN8FdUP00IVeThbtF2nU3txf\nlRTZqHeMLjOgIULjs7+odpBIkUEz3MoO9xhfWBn4/93hNHZuJ+KA4zkAfp4b\n7jvQPqkKSzLj8QX2+7Inlvd/A4TsCxYHQu+Y4I7t43tP3+nJjI1laAJ5/Tdw\nIh/aAD86hN+GHRb09MX/W9eT/PaSVP6yAe3HSLXbXX+KYBihc/Q7GAFWxtdc\nACriTj1xtGCdy87GVt+/OcWKy432aYVdCxTl1iIo6Qe0s/jbXKzw4wk30c0Y\n84TywpTfzBEuCTeLc5kBmfih5CTKEk0zGYv1tpkBP5KoxYuWxlsZbAixXIpl\nzuWi\r\n=QtOl\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20210205.12121c8.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.10","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20210205.12121c8_1612508875835_0.12985790834963185","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210206.dde4bda":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20210206.dde4bda","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20210206.dde4bda","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"19bd9caf1665d3a590a3aeaf7d06b1899eca180e","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20210206.dde4bda.tgz","fileCount":474,"integrity":"sha512-yq7Lio6KsHrOiiobWxeaUoQIbyDMfIWSqOJKJ6xjiNTEXzFwC9Uii/YICoyfsIPR1xfeEGWTjcykiYOfY8/6Ng==","signatures":[{"sig":"MEYCIQC5p1M+KAG9S0NNuqV8+Cmm5Z8kdbrWoas8coyKh7YeyAIhAORQQcJRTyeolO/xKjrqfGYHYA24+Kr8iX7wDfvdaDeM","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2376591,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgHkB7CRA9TVsSAnZWagAAfS0P/R/jaza3nxsdO+xF+tDO\nwaV92UbiAY4+7yEt+uz9/g26/mY2obq/peS9pzmRVHWcns4cYfG+MyyLVdAF\nn8VINyoXuxPSYrI+eLpLbvcK1ZyvleQq2gRlD8idOhyCld0pLuofFi6AQAmh\nvTS1vsuRwp8r/P/JYIxUTNej+4sbktGtZn5hQRsoQziEovx2sgdu4N/iYUHy\nlS8GO+oTt45fLLgzu2NZbXoGFWb6yb0QupigI0OXCjxaScv/ujz3r0AEuUQx\nthKCgHc4pJxbbkw5mjQ5CqgPqk5SdxEjMruxyzcjpj0FmWYRdwoeCk6Dp0pB\nrsZx6pPojo11cIpq1miILwSaWV7fdhKiNErkSYykR00g4BYHhLo0qpQGm+XD\nsmGs7p/jx1Dubk/jgsjG5e9sJ4wqLR6XsNh/7IFP+zXhdMo5Y5HztgSQtMUi\nhkKcmLAw69/+e4LX/Wv5otIkBbiEO+hI7oh/SpVA8dEIcz6/McFy6HvrHCGs\nohMjSbtRVde5lrlugqWXayqJFLO36340IrfI/uFntC0ka0nVxgbqEBS8A8ML\n4t4dyEXDcPZpuRoeK6ixkN91axMkrIGq0PpOaKSa/5U/qOSQGZSv+wSDvnas\nYKK3Taf1DEoAskFZ2eOYdfNJNMeAggoraD3xqCnDsxb6dbnpsUb79R6d+oIT\nDm6R\r\n=I091\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20210206.dde4bda.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.10","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20210206.dde4bda_1612595322206_0.5929499839112535","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210207.af5cc18":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20210207.af5cc18","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20210207.af5cc18","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"73994d1ad6f789f8835db4e477b36294e806a384","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20210207.af5cc18.tgz","fileCount":474,"integrity":"sha512-D6qz10u4wuf1UOdW2rhAZ+190aKRFDlxIkDQDNii+5hCYlBCcg+WGUUiN8+7okxIHAEWVCnFO6+FwLDSykL7SA==","signatures":[{"sig":"MEUCIQC92K6eKR2OElLU4hVz781DLjNwAZJYKN73FVgksboh9QIgZBOZl5v3SGCLtXnPn6HDV5LloHw2wk1VpvlSWF86ZR0=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2376591,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgH5H1CRA9TVsSAnZWagAAv78QAJ0kNNe6fOr1PR1E91Jr\n9dqhZ/FoFCJhiyoswFDs/d5JofTjVU63V+lPP8BJC08ScKaP4Z7RWkN3f4ca\niGUHI2Dv5aetvbtYJJyi2cKLlFIrH30DnKvryBE6KOn4KdjvhGmkG6vjNqMw\nNOiOwCfkZmmMm3dOpt7H9wTo6tklvZs9X1fOG9+Q+u+6VgLQQNPNdbiXu/wf\nzDyTNajDJLg60Lf0x7KkEunxWVv8X71JW1XZBMz3Mi+/DvTveheZBOrlFN8S\npfuPvMkX5EbCcFwOZ7I3WZbAzrybXF2Cocskbcaev8OMadr4bGIoXn/Xy41/\n1EoxZkRvjc3Bvg8+hG2i/nukfOhkOdkooi5QHdCGJUkUCKIXSf8jit/8/Ug0\n/xK4eIqjr7nnCHEMHYaI4krMfJyLSL3k+Mt6NLIkE9QahbCWv7diBogW2mat\n7O7251/dJh2kR5UZCOfJ04uppz7PELucJwFSvuMALva3vwsYgKtRIkptyXoL\nMgjX7/h6i5vS+l9zhQwspoDKHJ7a9QIi5DWZgq1vNAxW3FJlwdC7rwAAH5ER\nTk/2VMG8inVX/d9BQJSmVTYrv2vAe6uGaolAVi5cFG8hUiC7sJPOqQKXpuKC\nq5adiZoF9nDCbBPXXhg4OPRPaYkCxySD9+eYdthFdp/WRKEokXB/b5lR8NAN\nX1Yd\r\n=u6FH\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20210207.af5cc18.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.10","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20210207.af5cc18_1612681716593_0.6342420821992301","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210209.a44ee5e":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20210209.a44ee5e","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20210209.a44ee5e","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"e9c49c6bd57993b4e949f3dae6b2710fe7e1665a","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20210209.a44ee5e.tgz","fileCount":474,"integrity":"sha512-kbRMvfgNHiAnu8p64ogGNCJW4+CkW09sXQTKDnYG8a4j6Z2t6PSpEN41NOiioXfaMU7lneXfQ8IHFJ1ggUglwQ==","signatures":[{"sig":"MEQCIBfzetR5XZWRc+elZHaaP7NsSbxKT0gjAK44bumLjd0KAiAD4eRdFOgLPTJ5c6xdKZDmt2No+9qzqZlwkjKcLXMWDw==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2376591,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgIjU0CRA9TVsSAnZWagAAQnoP/28M4LAXMm/GK0Ieg5Pm\nuZMdG0zIG5uU05pcFBPiZcbzglgqEWM/Y3nfsewTXbkgI6xDgLuM1WUnNg84\nywECjRsX6NXDtk2pm4GpzK+uogwJyKFLY3cM2z65vdlOjQj5fz1+RPuDdvMo\nrbMyy0zCDDHLId1DVbogxxUhtYhbw3fFLS98NI6HehxhWkTcILgjIsHojQt/\nMOYuqb/SSOwhCOwKDHePLAnvtIAPTqtT/Sv7ScoS9Q4rM1N7xinKKVh2z1+p\n8Ia7nGdqCBorQjYAEd52jGCTyFL192shw3KMVb6KkOEHV2fqELg9usM3uAfJ\nbAM/+r88lhyyN/pr2Qhk+Fmo3zStrB7PIs8hZKvPTWz2tiJ1WolHdYlNSoTH\nY1zps2fJw9oO3qrQZpOtQPjRRyYoRBatKvTOa2j5uVj6+0CwFXT5cnEvbLdH\n+1lQ/mYsx9EFzJYtI/CqD4ZnXF3b3FgmpEjzHan7mI0dH29jEOLOHOey/1Eb\nLimwjl7bUd80bVsD3HXYd7bmzqoHNEqPV7l0i5pn17rnGIQfD9QxhzfXhT5e\n6cVX6E3xWkFQFmBZO3dGyydWjhIXlSpi4IsoyQ+FO8MAPxqUP4P/bvYQ/FtF\nuOR1WRxr5U7+w/7VbBPxrIbb/+8qbmrxdTR5l/WLAgQaRofxE/v1VEfk/ERQ\n2od1\r\n=vZ8v\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20210209.a44ee5e.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.10","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20210209.a44ee5e_1612854580159_0.24788915185675275","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210210.116dcad":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20210210.116dcad","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20210210.116dcad","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"c194b1eeb51b76dc1f09434288d951c5ceb86fee","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20210210.116dcad.tgz","fileCount":474,"integrity":"sha512-oKHYMLSdhsC2KijtVGzG/Bim8DkVi62VRDA69MbfUEkZBzXoapcXCsfH9cm3X2Y6K6xkeA+tlmejwbMiMF1SRA==","signatures":[{"sig":"MEUCIQD05csnWdLW3ticqu7hzPkOyopwGiuWU3eSQ31K/NdOvAIgW8/5NB0uK8jNufWlNA9W/PJmA4XwI7MLz7KY9AqyNm8=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2376591,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgI4bCCRA9TVsSAnZWagAAOdwP/13VjXoVu7o/PdfYD+iN\ntkPVFaGcf5CglhxUQ0juhMRnKm4Wjgt5xJsXN9qVgPszcsr2ZkfCIy5pwBI6\ngjhcr2VuVf3htIP5xllyDcl8OKpkYfXLWarXlzzumVBFYKdH+WjoQj+C1WUt\nP7W8aDXPKcnyiPXQtd9bFc/vCNJA7nrEqpbheBdVtHdF47bOqkcobIdI3nFF\nmb628fQZ9uqTkKXEbxvf4luMkOzRoD07i0bOpdoSh3cvKOhr8HnNsqBpuJ+m\nVqiJxYvTKDdX+YKF8uwbzfziIadiuAvVuSTKJ04SY5bXjOwW68ra3er19m8Y\n/mVNd41KOvRrEgo4MRLsS5cN2vbJ3TI4xOcVXQEotC1t+QuPVMNW0XgiPpqW\nhPmYQl+lRsM8+gXgAcY9s44z1VGrXRMqmqFZFLN+1R8s+GErmITeV7yIz+rP\n5fTZdNyebjEjUX1yXWOChKrPOSmBMKPgV+AMteyzwF+k8R8QsCU3tAyzqLHA\na9uyBWlX1HGGJzc6nvbPHq4D1dt/FdObp4G4WBXZl+VNUPQmN5+wg0Ev9Zcp\nF4lrX9IYci1BPkKruMzgCJc12sBfy4OnsEp7O8A8EeaaSJo5kRNOD85Ff5ul\nUTnD0EWBLkKv8vqbHGsqhBs0IETOTSho3jBub5Jzx1L/aQFpd12dw0ROzUwb\nMXbo\r\n=zJ1h\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20210210.116dcad.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.10","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20210210.116dcad_1612940993646_0.30812283260316087","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210211.c895ecf":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20210211.c895ecf","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20210211.c895ecf","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"612ca97512724ad8b2091da19cef3bf825cd1aca","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20210211.c895ecf.tgz","fileCount":474,"integrity":"sha512-Hf96va5TbkMDX2ORosQ45uemlOxO/Cp5j5QulXXWjo7IOohoxzFA3pBIYM6c98UoMVJ4fcueHJbOMOwrwyDzxQ==","signatures":[{"sig":"MEUCICbxwbT6N6Gb6QfrDq2/jRky6VTWt2zSxmTL2y1/U6dUAiEAgrEs5rtszQex1gYgDe0uRfrJbInjRn5dqcJy1n1neKY=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2379263,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgJNgpCRA9TVsSAnZWagAAcGUP/258/I3m8wZcpd2KG1go\n2b5Ot8RyS0XwcVwdTFQU20u9uh4oMgH1+jbk603dcaG03nXHELekyBhSXEgm\n0+8jVewaoGDp/xZwQemz7ePCIl6Jmu4J4iABl0eCgeUFrT7Z+1uv2KmBG38E\nRFiQeEcu8gME090YCMcLWLsCB+f+DmQXiLShLeDiNWEnVbVT29c6HGUbHOG2\n+xuyxNH3a3EO7wkjeFmRRrS7fTaIxUE30TQUOEI7e/+nG5EEEmt3i7iMs+tM\n06uYD2sPzThaujeg5zjYKdufgIJf75OyBPVAcIlQSLhpvUFEL7YKDqKAI5Yv\nKfgjzaFP+yJXlIQq59eEq/WYTmAI6EOS7OCAy91Cbd9JlXzIit5RLWj1jb6K\nfPuE5XHAROq/j2FeBbgvpfTJA2/MoBUfRH+/DWwEYpli7N6SREZHzOQIGvkH\n9a7GamEH7hEOlGkHcLNEiNSekpL9VCuw8zgW36YCW+JD5Oeb3SEy5qotngTb\ne4uRlBahlFcjX/GaWC8cU8KbCN4L8bTJyhGI4EPCu88n824JGhfjingxVkRw\nbBvK3JoYbmJFpULC6e1nmq8zJPPN54kycmCT0aS+BdVL2F1i2cAkB9vmvK4p\nJ8y9IH7h1PQ0i7ag/hHBsmVQ0Q/HgSrINO3vkr/4ObP0Ag/wQ4QNGuTmlUaY\nyOIY\r\n=hpdj\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20210211.c895ecf.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.10","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20210211.c895ecf_1613027368878_0.24094209576295222","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210212.2884eed":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20210212.2884eed","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20210212.2884eed","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"8b03e21ad6b18c66f37679d483fb358ef47ad49e","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20210212.2884eed.tgz","fileCount":474,"integrity":"sha512-yAF4QgRfZEOaopN7To3B3wTJ0ZNQM/6h1YcnlgynO5BmiDUTFM52nPU5X7ROnj5kNnVR6Of+47BfwaQmFYpihg==","signatures":[{"sig":"MEUCIFvMGH4SCGnXDLr25Me0GzjGR0GUbIpFypexWxnlFVpeAiEAp/tBqNyVY12Sxc9WMuNVyvB3P8U+0/L6jLpU0thPFTs=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2379263,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgJimgCRA9TVsSAnZWagAAgLsP/Ak3UtdOjycbcsfnlWMS\na2qaMqcPHtY+TwbRRrvIFFQkVEDytVdP29MFAdodtMtioSJgs9Os+1O5/+8r\n4PO9aGnMrhxAEVW5scGjOAnzUDZRKHT3uks+PHbHjEVp1Ffk1KoDo8j6b/eB\nHz1av1IDO4s8w9/e5ibEQcTUaoG40gdOoz69pBqLZCTCh5bYgQithV2igT/c\nSFQ9/iOwQuyvVRZnKrK0xrhbbnlCDu507K9qTaIwOx0hoI+9N72wF4jSLpJM\nuWOjyf1juMHd5T9g+GJBTFDn25/5xrLlrHPB2SZjiciG/SpzG4qoVbOUeg1N\nyj4dFR6uOLTDddd6h+CbP8Yh4pjiDkOFw5MZZM6wzLv7noOhf64VSvwfm9Pg\n95mCOVngaLz9bZFMwZS6Lkbg6Y1odVMJ7joSEslZWlbFl16kyvOxOKe2Su69\nwB113M17JpZo5dMFUjcP76UF0q3k6rD/HZrQi2ie8TEvmlrLFyYFtMPwhrRq\nz8dkWGWayygeWQlXpCkmBgwfgu8xsFSBgAiEpatYBvNYla3DtXuHjdqNjaNB\nzN1NJuhwunrh6OMdwNAfIp/JSDnPVKtZLAK2G1pITkPtRrv4wMmCSnCSxeX6\nz+ZbxTd+OPrPKzUNVSmJaqSns3GQzhmUgTUWLFb6xfjUUYg6qSntMg6ULLzs\neCKQ\r\n=lG5b\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20210212.2884eed.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.10","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20210212.2884eed_1613113760115_0.786705506877821","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210213.4729245":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20210213.4729245","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20210213.4729245","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"5ea0c50ae0fbd946bc78f06a59138fc49ca37836","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20210213.4729245.tgz","fileCount":474,"integrity":"sha512-1+VGEGrygr0NqpUPgZ+rxF+g6VOKoFZzVJ1CnIWPoIKR6LO0ymnvqQfAUVXM3yw8Ai8QL8ycr/ozSySmmFGfcw==","signatures":[{"sig":"MEYCIQCeO9oCKKJVrenBS/C7UCwCwHv5b9NHjDD2PHTo1rckIAIhALCryAqeDrfXrto9t5JuaUcwiMUUY9E8Hh3SOF43rvBS","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2379272,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgJ3s4CRA9TVsSAnZWagAAa8UP/0gDBPeI/oXF6XLdFE7p\nNE2msrv3OcydzavIswduzkBp4cUOByt5YeLoSf5rvIf8INiTnSY6TSqtYPYM\npBgMlg+Dsun5CNMCRgAYUcu8fAonX8Lwurbazh2Pe/4KGJYG9aQhMj7z0qyy\nEpQIRfzLYjo8N3u3sYL+oBiay6ROwMvhYW/ILicXL8TX37b3AfUBE97/gHvz\nGdwqQ8ng6GTZBpYxBgWkLiWQebGdGYWKKc/x7kBRIpEumQ6ofF0c5GNl9vIz\nAqpa7EzXj83l1Xr1mnL5LzU/as9P0qhbx7Z0W6wbnsKuHXLHX53Z6Cr8JeJt\nzHnx/qUiGz4EvJxwZeV1yw2gS9qP5eMbInW/07SB/YVmgQVQ6HjeTsbk5j09\nQmFqqgeZ4Lmk0BdbnEEA08vNQkBP0ZTwvDl31UpOYAqb8fN9Ip6z5DQwFVqy\nI1Js2tSNo/PhHoWbmKCkJpTYMZDl+b/zyX3XMRcvCHsVzy9iOE5nnJF4CcR2\nSIzFeWXsk2gVRHLZGBp2vl+Wk7SpmZmUG/h4su3lSPjEZc5KfM3JEsSbpQts\nkx2LwhKJjqG/DMCvV4/suwM9IcvQBA5W5zytu2GfXpApAkATy3lVqzubmhYJ\nrhHi18v0xEeH6sPr7RWDe55OAd1L9Z/33sbTrivOtcDA+oMvcv9Kgs05n8YG\nrEFe\r\n=g8GA\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20210213.4729245.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.10","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20210213.4729245_1613200184044_0.1960744671431025","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210214.41d8748":{"name":"@microsoft/bf-dispatcher","version":"4.11.0-beta.20210214.41d8748","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.11.0-beta.20210214.41d8748","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"fb5787029218711d16f463622a3dcd10d2ec93f6","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.11.0-beta.20210214.41d8748.tgz","fileCount":474,"integrity":"sha512-OMn6iN022U4Uqh9RB1P/IK8S5WOKpXY+3yFJgZS1k/nWTCqsDlN5uejlz1PcqtZL8pkp1vMJSini6xAAluYkrw==","signatures":[{"sig":"MEQCIALDBB/YHS7vzIbBvDdXCKx1HxVTYw//Ay281oEiuDcfAiB3GSILg6cgafQy+O0Yst44SBi3BWkYTvmG2XbKORZphg==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2379272,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgKMy7CRA9TVsSAnZWagAAiZgQAIlr2VFcKl9ZSVBX8stM\n+/GJUkAWPa+w9M/WLYFmj3CwuclYBH6lc8PoOsHS+a0qw1ruAjTM60K6niUB\nCiaAahd39Ml7591gnbK36DzUNRBnMhFVTSdgpfGcFP1u59neXsssUZZVpm+W\nP7GAH2/19ix09sTDacnFtYhMiRxG6YjCgnOALWAKfmUqhygLmugmyDakqUe+\nZwnDsqcHpF7fzGifi9t4czuu6PhusXEr/W80Eybdo3d83TVomcJAmsCTt7yE\nDjdN0iasTPC2R6tBgJBr8IgvheAZYtXPeogZokp7qCbq298+evW+/UIUg628\nr4KDVg8DBxnDIbGva1cnvchp9wzrzKWOWZut/nv0jdT9FIV+XyxUqvez54Ox\ndlEjgvSFL+eupKFn8VQFbJRMW0Sv6MnDlbwR3YfxJMMZPdxWYHKXe0raPOWA\nqIxCesjW9Lw3bLhssUyOcQS/g2LMDpYU3pD/zzt7A6mvvz/WhHPksAJs72zd\nys0t0dIdg1+3+r+3hXM7foy1sUMABq6IikHkywHHoramp2JpqxCzRWmqk6Wj\nTOOyxZUslCNc3m9ekZMetno1ENU5d5SP3XtA5fn3DLAyqeUdQWcHXJnIF8u3\neW7Hz7jpXPfPCq5Epf05HG05fI2mWv0oaRwx1vyxAG+oWW9mwxdT4uLFLFLV\nvZ2e\r\n=mTLJ\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.11.0-beta.20210214.41d8748.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.10","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.11.0-beta.20210214.41d8748_1613286586494_0.5135219926436752","host":"s3://npm-registry-packages"}},"4.12.0-beta.20210217.668cdac":{"name":"@microsoft/bf-dispatcher","version":"4.12.0-beta.20210217.668cdac","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.12.0-beta.20210217.668cdac","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"7d60352e98cc5c2b0500f4843ce24b1cf8f42c4a","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.12.0-beta.20210217.668cdac.tgz","fileCount":474,"integrity":"sha512-7Zjv+4QAKce+c63j2lL++ZqOrpH6etd5ZtIMPULRkRyoXbcjDLScHPC//cTVXqu9k9TsXqhH+LZQA9RmQSjkSQ==","signatures":[{"sig":"MEYCIQDzEYvFyXe6MaVodlwjMfwqQ2H2V1Aq7PLSnqr7x6obIgIhAJPgT+0q8EC2e9gED6TvI1pWDA4Dx7xGDQIeDCA9Cvq9","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2379272,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgLMEMCRA9TVsSAnZWagAA5GkP/RvNq7+kpF3H+Ni0PRC6\nATQmzBy/gux0JA5eZO62SM6xnWZ6OUxtB348WGi2cAvsuBzVGTsHyrdepG3p\nfN20FfEi64sVPEtu5tWnBxYjb91oMc+4mhVL0tMC26w5iWJtXHoz6GTMGTAi\n/am4KOYrr2PHsUFcDnht/MN0e+ZF9XT5Kmr9mAi4IloFjdHumWJjxi60+9mg\n+kIi55ldfcqKmnlaClebV/+Cgg/MCHaauzQhAb+TWi1222vnKhNa0dJA5Rte\nnw4dviht+NFaHGfK2Uf2eg9o3nxdB0oEmcAc3Q7w4KOxmKuwFR6c4sBcunJq\nJYAMZzyZZG1uPfb55Z0GJO93hZx+LOq08U2VKgQda25vkRMUbo6envJPg6U9\nAL+55p0mG5HeFMNF0m/D+TMeiDaVxGfc6IJ5UBu3bnpueErdzjrUYP/u7m7b\nvDQZ/UZ7hAEBfuFpgII/l7MIEQPTerkM+Cz+y1ZrPd1nppvnQ/jnjqqkMKj5\nwr33KeBPWWLDBhqlB+WljMSwcI7YUIHauJ1ZnAhELdwQwCQ7sGvelWIO7Rja\neXejOjKwoWt+ds07wxmd1/8zlgVA8jn9UCccrSNg4xnPHuuzC58WLK3Db4L0\nn2Q7/q5B2hqvykluaQmfGduZqTbPKNHzz/iSWKHjD974Udr1KZ5lEUppnLLJ\n3dWZ\r\n=9ghD\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.12.0-beta.20210217.668cdac.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.10","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.12.0-beta.20210217.668cdac_1613545740242_0.15154578120485507","host":"s3://npm-registry-packages"}},"4.12.0-beta.20210218.668cdac":{"name":"@microsoft/bf-dispatcher","version":"4.12.0-beta.20210218.668cdac","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.12.0-beta.20210218.668cdac","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"17188e85bd0fe14bf1986f6ae5d13249d57ffa12","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.12.0-beta.20210218.668cdac.tgz","fileCount":474,"integrity":"sha512-kJYdLgMB50tUoxrDUOMfZ2tuJZirga43KJh05RZoPM3lG7DD3GIVlU4fjUtw1juXjDoEIXB867WVe2XefHi9og==","signatures":[{"sig":"MEQCIGFoJtIsIZuW0SAEzBR3PfegBr5RX+nCmllg1MGQ2tQNAiA0SScUBLARGiKARHAbVkznBO/JUUliTLejZOp9sSSykw==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2379272,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgLbhyCRA9TVsSAnZWagAAdAQP/0nwP2P+xM7ZC+K4vwdQ\neaFLZoCrh+6zV44aRiu6yG/zJuzFfvZxgxCmUXDIuv8RbGf9szBBvwY/PNg5\nrNCy8qmNvs4C10J9aPXv98oMTZbVhirWA7aQrpU+KCfzqMSp+U/wz69Iw25K\naF2HHmmqPpplWMFRpe6o18YCQQBJeQxS57JSNC1BNBTNsRCAGnNxMrW1qAXr\naG5qTG9v3EvnvC0YMDeyCfHHLMchxpMSX+jTYW0Wlllk2vrAbm3oEKK3xqbJ\nGMKPg3h9f20LpVW2w7p0UDC7+zIGMPnC8wZL0VflbOhs8lWlESDVR2en7EeJ\nxGDjBb7IAs52SXoC27XOB1kufuOGAnCeMpml/mfC7PrUUY2HiI9kyRxVVmmJ\ni1OxG2ZDhrTbyO8WeEaTfNq18KsoyhbnlgAMTtk9/tKEKSNik/jIuCF2sbRQ\nOA3GRedcR0xSYIM3g4qvfFudb0TPqu6pCmKOqS1nudl3l1wpQ5+Ahwupd43j\n33LbEzQLrYifwDTPVn3IsuaYVzlg9EuS6BsFZcDd+W7siUWfapHCU8SdIbvZ\nugJ3PAgy5krgBc1RndKqP/RmlOlLkrlFVehYURxFHCcXLYsgfbwTKmaid3hS\nfdb/jPO2jHU2Kc6UbZDTz2Lf9GM2YQSuI6+TUeNhKXARRgC37O1VIVv+FWoK\nUzVy\r\n=lPes\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.12.0-beta.20210218.668cdac.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.10","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.12.0-beta.20210218.668cdac_1613609073426_0.9166572035968663","host":"s3://npm-registry-packages"}},"4.12.0-beta.20210223.d42fe4c":{"name":"@microsoft/bf-dispatcher","version":"4.12.0-beta.20210223.d42fe4c","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.12.0-beta.20210223.d42fe4c","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"02d646cba4fdfeca0c7a2225b5c488b3f4d59074","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.12.0-beta.20210223.d42fe4c.tgz","fileCount":474,"integrity":"sha512-TAZQkQjhVr3tljp7yOctORl9QEpx/rR9gzCN4fx/YOYRKg2tJWEJCxGTpy9B6cEem02Mn2fFqz0hkVFLRlJLSg==","signatures":[{"sig":"MEQCICFzWg8f3tY8nCInIguqwvPvBgCHxTAgqR15t6yZMULQAiAbQvFpsH7YACE3bwF0SFmaHtbsSnRzABSlQBP2wnytxw==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2379272,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgNVemCRA9TVsSAnZWagAANewP/Ra2Ka9nN6owJHRVuDII\nnEpDbqajENlGyXaxx9cn5/kr9YPlGqqOClHZ2b45kdyDopqHv6K/rJmnFpHP\n/T2ynx8SwNtHn1W9sPoO7+yp3qzSm0WlQPP/YeBicZSK0JonK4WBWGNlyXO/\nl3fP2EMsUXolT1dvnnoI+pSHb81KDtwi5tiUl2tcmD+a1QYMDcmW/natysej\nMcwueTIs2FgCJk7mdJzLIlHkYpyWOBYuskaCAvLQYub/Ens1xmXknoHYPu+4\ndFTuumFEEa/De/DM1ak7PovJ7gia9cZEgNhvm6DokZnnGrdVsL/GxE1vv/GD\n+YlVUnOY3zX2N9Xj5t9f8YhpYNywz/vJrqfmEc0XvXAsOFfcCV0kZkRnK0JT\nKYDCtvC5hcVC8trF35WefjjWhkpm0HOt1ez4qbGAXrBqlag+n/Q/fgMnRTe9\nkZcnDKirkOpXTKbG3G9c98zf+MXyLvuzYbDGWXubjcnV4mlRF/2QulYii4iv\nk6Qfe3lTzRNJn1baacRa68xlr7AwNNIf1jn1lkFuUz2HrTcym/iyIUrxdBOm\nyxuHUMAuHeu+x3fokD18IEh8eIwzbVy+oTAIWHaeC2FlerU7tKdp4b36R3SI\nu0+Vr8Rf3zLfV/Lw+fJ2B/kmzYtV9Pj2pXcVcBHg452yirwhzyxirX5YVNLx\n6xOB\r\n=d8Vj\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.12.0-beta.20210223.d42fe4c.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.10","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.12.0-beta.20210223.d42fe4c_1614108582177_0.44951405829665436","host":"s3://npm-registry-packages"}},"4.12.0-beta.20210224.8610f7b":{"name":"@microsoft/bf-dispatcher","version":"4.12.0-beta.20210224.8610f7b","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.12.0-beta.20210224.8610f7b","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"65d1f63e361efa5788e027dc34bed9444cd36c71","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.12.0-beta.20210224.8610f7b.tgz","fileCount":474,"integrity":"sha512-mmx2eCeqLYF39BkFwhoSBjG6+KC2igMFCDvwZw5lsm0QuqHuXTUKnmX5qLV8hCuw1fW4SAFRn+XfaAPrSjjoeg==","signatures":[{"sig":"MEQCIAv/VSfzVJsjkmKldB780UYKZFa7JkT5fEfL6BOikv25AiBPJuyE2rUJNkwOV5pTjxybKEHwBd+xcAeZIsRfqZbMmg==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2379272,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgNgSCCRA9TVsSAnZWagAAPUIP/1/xHd/Lz7/ZZclijRFZ\npiGlfQF1NWsQidcDXmgTN1yeEG+pD90XqdoN9etlor5LrQUltE1pqpFLdjJV\nSg1RzEG4eSFLS5FseUqkX2c3TIn36vPCIo0IKlY0BBDyjTFVfr52LDHljQah\nnakVpHUG36YRzql15t61gatG8aPOcbcqKvYIaz+ZqllzimFsmGc/isw3A9N6\nzYamEnSM9Tbso+IIiECVnuxIsqe6npcxG+IjEr+J9DbYXlSWEJlhJs/hBFQk\nOvTzx/pd5Ijf5rMXzh6azFuA74ffVZ/uVRWEtlX2PDU3OHQZn9wPYZGUxraq\noEuhVaDE5ApI6Q08QMnQLzEV76l9963mh6dtCGFWvyDsL8Oft20zkIr+2FS1\nbJEd7o5Bs1kF+kV0Sw/JckKAWa5vZzNhRy8FF207bUxFMoJG6tE6whHwKV+W\nhxWnZBVt+MsIn8BiAeuV7aVwQLEw97rbB6qRoVUHjm7WZcWvsODtCeZl8uaq\nMcEN35z2k4BeoYEBpRdFgpUe70+NFmnPdOR9IC1kezeb3Ck2DCM9lclJJ7RG\nUsETwvySLi8JAx+z2E52aqQ8wa7FIpjq6HH9jBUz2H4BhSw+9jlcp5SVTonS\njkFHeCxFxKth4QgCQv2lYitkjoC5dN2YIl00Xn+2sX1uPZbbdYe4nV3gTumZ\nhgrr\r\n=1swk\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.12.0-beta.20210224.8610f7b.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.10","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.12.0-beta.20210224.8610f7b_1614152834096_0.747466756116977","host":"s3://npm-registry-packages"}},"4.12.0-beta.20210224.dd555e8":{"name":"@microsoft/bf-dispatcher","version":"4.12.0-beta.20210224.dd555e8","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.12.0-beta.20210224.dd555e8","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"70d703060958303a016a5962509b367fab5a8b14","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.12.0-beta.20210224.dd555e8.tgz","fileCount":474,"integrity":"sha512-td1d8uZMPMlvybMd2gKPuYANIyXa0mu3H1zGmd2R+I4SgW43yyWPYcSFYh92F+ZhzKnBieBsvaLxz4aaJZ/CHg==","signatures":[{"sig":"MEYCIQCouQVOxpDOn+5+YGEo8WIULrgzZ2yhazW3XyUAHki30gIhANXNss24lua2Sg+aYLMIxRKWfAm1mPMRzXVU/28RqUHF","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2379272,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgNuD/CRA9TVsSAnZWagAAINYP/j4hunFTzpGlS+tzLNUQ\nbQpDEaNIFVCAFimEYbLmBvFjVmWneXs5Ga9mcQwCBvL7I951VAhJMFvuXnjr\nQcV/1zcX/+vfIorRMUgvHKaioUJvXt8r9q6h7NMVL+aQoBIv9ZEW5qw3Lmly\nI2BVW5GIQNo7rCLdo2lZk0OQeEYQ+Cu5OyLxRVQdlHVUNqcbxkQdIF1PDbcn\n72WBW0NVOpBlrjzF7EAY1K722pubmYYx+3sLMWvzFxctCNomCDjaG0SolCsF\nCi/vnu1WdIkEsjv7vH/Gg94WNthxdWvyFDyzJpTHjp+rNWH51Gb9btbcnD6P\nwgunu5y+l488lfbV5c2mt4RvYS1JKZHSbEp7cxwUcM3QPFKq7inf9F29unL9\n11vLXpzc3kQ9SttIBO3b5vhaR9DQwretihKZLSy1SY6qFYa/v40vi1tUAJox\nO7oe6HyvKQy8WVO8Ebxd/e4UyPAnj7A3fdmfNeizB8CIRLyYiOvq79UZ3Ru6\nE9WtR1BtVzI0FmN2bk5fWpmdHwnczBBHCT0+6Lyp2fQPCjbJGuEYd7QSCLkp\nCf1EQBOQw1RHqEbCiOABaytb4JpXxNK4Rb+S6hP4m145ZHB7R1LNDS2/ppUT\nY/wRuNpRxsLgwAZJuX7WgWqxaXkOk56nn0qgMUWdsAYsM4UhqE+0La6WpbbV\n/0z7\r\n=9WCz\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.12.0-beta.20210224.dd555e8.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.10","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.12.0-beta.20210224.dd555e8_1614209279082_0.603584312000782","host":"s3://npm-registry-packages"}},"4.12.0-beta.20210225.6744890":{"name":"@microsoft/bf-dispatcher","version":"4.12.0-beta.20210225.6744890","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.12.0-beta.20210225.6744890","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"8c58fcdd40d6488f148197cb19cd04cd45e32651","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.12.0-beta.20210225.6744890.tgz","fileCount":474,"integrity":"sha512-ZZ0kk0BcrCQX5kWowR7613O68BhOZDkKDtva2s62Ap1Fv9hTPlMDZgMDhRXamyIbKREiSIUA68l4TC4dy/8W8w==","signatures":[{"sig":"MEQCID0hDis8eWb14usJk8hiWgWBHa2akd10KaD40SFGfXl+AiAQr3lPAQayQJZ9/XdTOJsCOGIwCufi4y/xco8aaqF3wA==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2379272,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgN00wCRA9TVsSAnZWagAAlf4P/1WWMPH21RAdYhh+e1Ob\nDWFGbA3poE7pGtyoeUtjoIxJndDEQ2lgIXTLKTsovhh6eCG+aB5e805Sz2Sj\nqYm5688lLbpwuB+nJyp7sTUh+p0JIuAGTv1NoSy+B4lPRT4kR8mgkhrKg5Hn\nNJCtnc0KSmbmqvaTY70QzyV978lmTtVH+3oLaEykartOEQtnaJ7d+l6+Xn79\nYAkoajfrH7zpWHQ9B/uOFu2P5T4QN38gFzd3L9PSOG865cxaEjlc670EqzPA\ndLMUtctd3/Oebt+1KCvuWWNtoj7UpeB7FM5UIdjjgHmc9I1eZk1x5SqxXPsx\nQ37PJp2Nm1drI2q+8ak8wivdIeZVpNBrIEeum3+8SFAkio72+eXgcolpHwZ4\nexPlFhLLfRV8+3Imj4+eePaKVNLw3cgETsw9GHQ5OoDw4luf6+KLCIO40sXt\nyiSfMR6UdTzpaXz8zYaknNyZYL2LbEtiHOfnuSbRxhXPJXtz09JKC0bMuMIi\nJKgYgokfmV9O5ACUHHRx4+F3V444+u0Y4T8EB2qyZ4f0HVjq0mF81lpNpshM\nJDEE1nbIYMhhTBj3pEQB/o2Tz6XiC54euMP7oNRwxuuYzTme8wVP2uAThLNj\nkhHkEyoZZXgX4l/N3+K5jHqbUeMe+oSQ+xMCGv+WYlQwj8MhTfd9bgPTWLSR\npf1Y\r\n=ki6y\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.12.0-beta.20210225.6744890.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.11","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.5","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.12.0-beta.20210225.6744890_1614236976320_0.9652953675769094","host":"s3://npm-registry-packages"}},"4.12.0-beta.20210227.ba15791":{"name":"@microsoft/bf-dispatcher","version":"4.12.0-beta.20210227.ba15791","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.12.0-beta.20210227.ba15791","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"3ffd88383c36f78a98061bbe8e053b7824c881d5","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.12.0-beta.20210227.ba15791.tgz","fileCount":474,"integrity":"sha512-hSukWILe4sB2Gf88FJpeLzVhPutL0p+N4PNmWqn+iSbOn3ZfXBVVlX0bw8xX8Q5lQs/EmY4N49x//dJohCGWUw==","signatures":[{"sig":"MEQCIHPbVn0n3kJCCRrnbxUn6CVQ3IoesjhHNaU1bUbTweFPAiB9IPMRUb3lErLdKQx3BT1H7dOJ3sHXL/Qo2sfa/g4tSg==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2401292,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgOfAKCRA9TVsSAnZWagAAC1oP+wZO4MtIyTiN5FwqnLf6\ndBHP/S8ytFRqfahqnxm/oswlQRO6ORpRzB2tiLU7BHQ/9xEI6yYQUhPeucQd\nvLZJKVuzwQ+PwnIj3yRORCfhomIT9RRo8xnm6ZdZDMAWmIPMkx0i/LPVTVo1\nc7ieQZsunSIcqSotCEfwrzdTHPEBsRn6nG6Bz0TKmALhkZtwsJiDQMY6S/Ek\ngp3YEgqxx7174S/YcRD6jG+sJH3XJ78qdfGEs/yZ8Wi/h4LGKGbYYh5q/CMi\njPUv9ND1mxSD3VYmcPUNSqCtFZb5AlzkXSbo8iinztal2j4JnJq/uJEZyaid\n0Opv3kYcF2/aq6XHpxSCOEWoC7prjEaXUtm6hXNYWZlSGwFd7g4gpP5pK7wc\nmRnzYeb2WSk57iJfajFunXS7ty43YU6yUL2mSA6rSSiOjd6Y+3E/vrufbG1G\nxqgJKQ2DLbE8w203EcB5rbNKQ98PyrC/HCfUPpr4VbGSIEdYvUjbEfWyKIU0\nm/8sV2/vvbHX4KrCW6Ynh3RhE+ckROAPckb/KLWxEzPluQEl3as2UK9UfXyF\nDkrB6l6LBv+cAcYK1UJZPOqASAA0yADFEHqbQCBr6rSsfW66TghHRsZVBvOJ\nS2M7vsSpoGcfTQZWOaqdVJair0yv+B11B23U/bZcX08ejye2l0ApLQWhS+8n\niunO\r\n=eV09\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.12.0-beta.20210227.ba15791.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.11","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.5","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.12.0-beta.20210227.ba15791_1614409737360_0.83897851003026","host":"s3://npm-registry-packages"}},"4.12.0-beta.20210310.a728f41":{"name":"@microsoft/bf-dispatcher","version":"4.12.0-beta.20210310.a728f41","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.12.0-beta.20210310.a728f41","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"dae55877fd96ad9e2f89b4040a954b16f07c6990","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.12.0-beta.20210310.a728f41.tgz","fileCount":474,"integrity":"sha512-12oY/8aznIchRs6TmwR71uZ+3S/eJkbceAy7l3uBGXJs29Kqta25cvndtwxTMLLBg4QPhPKUXBwvSIXh2RDNAQ==","signatures":[{"sig":"MEYCIQCHhllJted9hoWgSv89oZBo3mqXrRrXW1XAB3VkPg2+xQIhAKQajCdqfcJhKZw57ZvY8CyVEGTi4AB/9JAheqTruueP","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2402309,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgSQS6CRA9TVsSAnZWagAA2ysP/jd/OmEkWfT6TSFuU/UN\napv/3JcznEz9ocSeAi5D9nrvZda7CjQc/4XJLOB9WdLwgi8T7RuykejToRr/\nmj+CYq2iu5f9y0uZLTjSrF6wG1aoIPTgyDBcV+sIR0UV3nrGzK5l4KQ4K9yE\n6ooN3P5o0OPwZ8tjg0tfHW5+t1eUGfFG6bpvo8zAfBTeFKZMmcaw3yd/HV3t\n22OAdK7ccFcVnkQwiU1zReC7AXSpqeJ4+5B8QmNXFxXEuMiDTqNR2inTsUe2\nlz2Obgp7yzoyT9P0MO4qYFvuL5KkYFfUbh99253OhQYaYQFpTXkIyzUXpN+a\nrFhq64n0cqOWjwx/GfhjPWnCHHuxUNvlHjjziTZ7i1UqJSef2it6jtg/COLC\nwzS5Gq79H8zoWsX0Mf+6Fv77AXps21IAGCF/+2JtS6CR4qIIZgEJVRKzA8KW\nj+2LLLnM4GefHjru3+wetCttWFTPF7DjJQQFdhVbpsW3SmDJ+0SgBCpVmlRg\nw2/VtlJ9aXfrJG2WVt/1y/H8UMga0ZbOzgxhXOt3j41Ww2D/PwII1QKIlSHq\nyBpDFIaTcmm9NX4AEbaY/21/tkWVfU1sngw5+J8mgCkYx7gekw71fxZLsgp8\ncmAtaNbXGMY0KcmMdo9PdoaN/94+nunV7tGeI77zGDNz4zI35i2DH+ihV66a\nINzK\r\n=caP4\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.12.0-beta.20210310.a728f41.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.11","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.5","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.12.0-beta.20210310.a728f41_1615398074028_0.46941490726688007","host":"s3://npm-registry-packages"}},"4.12.0-beta.20210311.6018ddf":{"name":"@microsoft/bf-dispatcher","version":"4.12.0-beta.20210311.6018ddf","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.12.0-beta.20210311.6018ddf","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"e369780c48ce0f01c39c8eab13161d75cb94476f","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.12.0-beta.20210311.6018ddf.tgz","fileCount":474,"integrity":"sha512-L6WpLjka4BEfOkmmI5wV98FmMPcPY22iXMxizx8+cpvcToJpwGlS+UlP+RHt3uSDwW8/jsBOdo0WrEUOdsHgvw==","signatures":[{"sig":"MEUCIHBie1Xnknw4WzZZ5ZILM9uxXD0S03X9D1j3Pg5moK8jAiEAogNi0DcBvmRrCFfDZiEXinUhRcmHUQqmlNzd17n1t4M=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2402309,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgSlSNCRA9TVsSAnZWagAAEdkP/05CwgNqMSVcSRzuyBRa\nIjx8+5BWDW13pZFfZ3wmtXhHBI4zGHcBqMNps5eAv2xjcFW6MhhH6WgQD5Et\nGnVqUsWYudMe0HtIRFun5viXR3zlxUQcJcQphbmyKohlkXAPHQZ+CvXjolb/\ni34kulMmy5r6U4PhHoFYwieGsvX6OhZ5FQ0iKwaAMOPynSQCyo0kCazFndGx\nhYuJnK0yRuO8lmuK5eeTJ/d63qN0fkRL4mh3zipHpGiEcZHX/0W039uLI/mL\nBj/XS9lO+p5vjOPlUwOErNYXNXfWpG3wh1p3BnwmyTV+mH90/1KcfOPFRcVe\n3sxWsoc74ASDpeZbQlTQU39R7P1zUsHV0HqDcR5RIK5Fm808Gz8jojUdLjYd\n+xtEQq7L26MP4X2n2LG8fkHtXlz4fTQQ3nmSMsbERYWLugCQKFYWPdiiYIy3\nBuWcdIzk7Ygws4d8T+4ihZvoU/XfkNQANm9uO6r1+TNX9o3s6ebO5xSOjmnL\nhaO2+0/FExSnPX60VXxAz4IaZ+PSndkhYm8AAhoNxHpCVO/19rFUi9eOO6Wr\nLKGK7f8W8CHa02lH84cgk1kK6IV9LsgYQR6w3aJB/fthGW6GeKwHrEo3HmbV\na3DNRIHD5QgQev+993jdOyVSABIe5hk6pGOE6IQsyvq8vMzILst8+oczXsIA\nPxZ+\r\n=TjJP\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.12.0-beta.20210311.6018ddf.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.11","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.5","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.12.0-beta.20210311.6018ddf_1615484044701_0.865837439829217","host":"s3://npm-registry-packages"}},"4.12.0-beta.20210311.98dc244":{"name":"@microsoft/bf-dispatcher","version":"4.12.0-beta.20210311.98dc244","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.12.0-beta.20210311.98dc244","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"f08801621d16e32860c6c79a3d3b7f0be75a4e51","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.12.0-beta.20210311.98dc244.tgz","fileCount":474,"integrity":"sha512-GTtX9cl7pfN8YDSIElXbM02bsjihDrYsV5CQzQh1dYV4akDiOUF14yBayx2fSmYdPGZQvRUgYvewSYJCMpzATA==","signatures":[{"sig":"MEYCIQC0yGkPblF0A5ogFKdmJhmuxcShAMFvEcE8OTjeGBpxQQIhAKoMhJS88PXxfibIHl1fc9WcxhULaIfQdJtspMcnDkgF","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2402309,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgSnHtCRA9TVsSAnZWagAAuEMP/1mMfKPU7YEGv/1Kc6/x\nR1mjvLXuYtC0DHooOxlWIqofSGU7LPHCaSSU05CMqgMCKp8jEFa2LXoOy/28\n2D6cEHyPPFYPVbiut5RD+vF1o5vph/NKKOXcqb64ti8wvOYeRbv4ezjyn4jn\nDNA7z6uNWXWWITIxNwhq2nZfbBmQ2XvIRyQLGuaQxb8SVLoY9RL3/UdTh6Vu\nvLKxuZ8zVFi0IWxCeqSQBlou5dSzJvZwD5r5omRrjGUyEcxE6M+8rpVQaDjI\nvDYEYUhyBIEWep7dEOy/C9ikg4SPrmKQLyQStBQUrc8E0mx6XFJKtDnc8ff6\nL8MNVW3tz+WcDm++h/vTy1t8yAsbRyBR+OIUkPQSzzL90zJd+PGY2SbvdVHx\nf/+pGNUX8mxfm8co0LTTF9599Yh6wlkSwaqgb+lhLK5hBHrDntpAqGg0BCIW\nmGHJK8k9+wOp7q5GFwDRO8PH9rv3bkcUqAJLhgye9fu95HTj5tSZz8JFxuc9\n+ng1XcfARyJ1mHIAC9tLqUsfXnbh14Cz2hpKHhZ0sVvn8TnCyTrfKYTGI5fP\nNPjeOr321nnMQ8W8dwhhcdgrNS9AuztnBB8DWn5HVEU+fpP28ReC7W3p4mPj\nT7HuPNA3l059jjbLLcNYWKhz5V1zofy7m8ARwBjr8HtqxT+6x0VDwZHCFGzz\nzl39\r\n=8fYn\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.12.0-beta.20210311.98dc244.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.11","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.5","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.12.0-beta.20210311.98dc244_1615491565294_0.3195474418263904","host":"s3://npm-registry-packages"}},"4.12.0-beta.20210316.cdd0819":{"name":"@microsoft/bf-dispatcher","version":"4.12.0-beta.20210316.cdd0819","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.12.0-beta.20210316.cdd0819","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"454a612d4e17edc964675259be84ccc1c51425b4","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.12.0-beta.20210316.cdd0819.tgz","fileCount":474,"integrity":"sha512-TzkkoUTIITiBjwB8+UIjle9PCRJH8jolczOTEQszCDB3t7twfs54tLXzTmNVBS/bP9EAgZ+TY4xzd8IhoK/XdQ==","signatures":[{"sig":"MEUCIBI3xLW1r3voaI3ZdgNjpHsZzw73bAWAAyZNV9djrUB6AiEAxw56w6qRuMudJQUQhUPulzWUAohTzywU139lwSvYP38=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2402309,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgUR4qCRA9TVsSAnZWagAANIIP/iJ/bevibWRxcpPdTj8w\neDDNd5Tz7t+q2gqcKl6ITQ72BAUDsLTAaFB5F9lSWsg1JNHMlAm5YEj9gBKT\nielxOZJN8WN8UFDBkSp4ohDSTKaIuJyAM5jKsNfuSzvm9GskeGefnK5wL/ya\nhIoVK04iM2s5CwEiC26byQMIR37dkF10g4NyH+MizmzT9kRkra0YvgdESSsu\nRk55U2avantlelsz1weTNanHIpESvpB1Tlx381S7uOBvAkuT8oYL2GpvK8Ng\n2iIE4B3fI/wpi25JHVpEF8ZcubLl1BeSegZxPOn1hH1mnFM40YHqoi665Xjb\n+bYXqyWYj+PizwEsViCy2lKjvB7gjvCn81mhA4TQUJkPFA/nAdPfVimfWI51\n6lvQXChzHxxKNO900GwPbneeifVftfvLKoN18o3YFLRqDaAOv118FQCxaYhu\nyJCO47hp74+pqbXNwCzlLAu+itcCHe8VibEhTxKNOEXXFebZ1+P1Wy2rmEro\nfoJ/zth8MTWptHOYJ+gz3KiYaY06QAgjVCQ1oOP6QC5v255fqQabg0sEI51L\njbry/FfKW61AdJu45p7AZ1F5ZJZGwZ4RmixHDyD75VqYFPie3KIol6yaBMax\nPWNMPl5VzFbVudXH9oVDn53v2nLHpNPyYtTeiKxlmdRweygPfgVW+fGqoIkw\nwQqh\r\n=dB5B\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.12.0-beta.20210316.cdd0819.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.11","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.16.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.12.0-beta.20210316.cdd0819_1615928873401_0.2241921321814464","host":"s3://npm-registry-packages"}},"4.13.0-beta.20210316.e8ec340":{"name":"@microsoft/bf-dispatcher","version":"4.13.0-beta.20210316.e8ec340","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.13.0-beta.20210316.e8ec340","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"de2024f41b217c0aa937807855ff519ea1dc370f","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.13.0-beta.20210316.e8ec340.tgz","fileCount":474,"integrity":"sha512-meLDe5MKbXdnPYBy+pXHhH6k/cdGs7zW3Vw9Powf+OfX+oBEgxo8xlfa/J1rIBsleIiGWU9reU+D6gKK/16Uyg==","signatures":[{"sig":"MEUCIAv1Oafuve3NR0Q9WFzTosmCdl3f4HKehTvuvVKl11nHAiEA3OgIetM06qEm8AyRROhlXGhB5dXn02WIJAnwKm8/bmk=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2402309,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgUS6pCRA9TVsSAnZWagAA1G8P/ifFo3DKbooCPgwjqpTQ\n8fNYeH3Mb7/t9x8KTQmrPCY37bz3RXy3+6Xyko2rzDtF8FC7P/2jm5c2kDG7\nQ6oaBdkBCOAHBcT5PsCKBdjYGrdSTwjT5peufxFy03TIvQ4e1MAijKUSqA/i\nbWOHSCMqALL5jbr7JdxJSOCup3Ut8hsRHNcFrHtbod/nJTJaKGXUauLdBFc6\nrGBxu2ff/xLaVDTVDkUiqg3aRBMXqSHr3Z9Gyj6eNepyVqhtoB23vLIy1BpC\nxqf93u6Vx1vXwR3NmQG3BQqaq3FnpcpGls1EL+sov95p7m5OCkhMZwMzQjUU\noPkCxUc0tV5cgtBLiVQJV8aR/15pyi8SdF7MTKMSpnF+IVugKZOz1E4noBXD\nrGyemhj7IqkGNdnrhtcX4cn8YfnSZMpvLODOWJmSvg1zrRWClXvC3chwioC7\nynSjKmcBteTR6gs86AtGszMyJMvTGajcSyYlLoq97wWhGNgsjpkcg1zkEEJo\nz/wEs51Zu/krHmkx03EcDtn8RGUxEEdQuzcHS0vesi8UBm9m1bzvGr6ur9++\nat6D9atu6Evhw4bsTRSUKBFwijS3x2HvqekaAyGMLTzd0MG8uWt9SXzoMdzw\nzVMcv/MWy8bDiUJMo7JQFer/qOhjWKOl8i3vLgJdXBOOysQtcntJbOHLVMB0\nXPNc\r\n=0AdP\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.13.0-beta.20210316.e8ec340.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-dispatcher\r\n========================\r\n\r\nThis package is intended to be consumed by other 'command' packages in the Botframework CLI suite, one example is 'bf-orchestrator'.\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.11","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.15.5","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"next"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.13.0-beta.20210316.e8ec340_1615933097307_0.2980570945088692","host":"s3://npm-registry-packages"}},"4.12.0-beta.20210322.314475a":{"name":"@microsoft/bf-dispatcher","version":"4.12.0-beta.20210322.314475a","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.12.0-beta.20210322.314475a","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"48c2971b45cecc01461440636cac8fb74a5aac55","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.12.0-beta.20210322.314475a.tgz","fileCount":474,"integrity":"sha512-f4LrW7fRLmB+fZXB4OR6bHbKWMAY3E7tgELhSq5fziWCX2QGsOfD7iKekFtFT1NUXFl+J02a0nQm5O8li5/Geg==","signatures":[{"sig":"MEUCIQC8BC4N18YYomZ9DiqTQF1SjTe10/LeDTwv3s/KSDm6AwIgLq8nwEeD+Dp/r6/IdE2+O9g1iZDNTohc2BsWdY87B20=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2402922,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgWMePCRA9TVsSAnZWagAAYtUP/AyYzYdsdItHtlTZQxEw\nPxvhOTcsB6CJbD0RWY5iMYmdWfBUBSczZ43ljYMHZ8Z8SW2WFZSjC79nkccx\nhj3KQiGdDUBP0aCzZ5nM65mwCDPh1zDdqK2ADCHkWKRCUB9v4WaqxaSF39U9\nNQR4dwgz80v9o7QwC2p4BG0wtHEinLCFiQCiTDMBW1Cap0oWOE9OVzBQG3iJ\nbKRSxOrq5LZ5A4rvFJ3EbWJ3orljKJo6jvcwVB0vwIjsnvm2mLVi5Nrlkk9M\nePEsl8t12VvPTbYN4UY4fOMz7SUOweB4Y8ZG9gooqHqQU8Y9dTKz1U3qWD5J\nlFFgMROu4i6ZlOaKTDGQildHOZZo0rRf51SqFeYAcHpwcmyJ3yblxylqjXEf\nQCfBvsg2gCgKl/52jhvPpR12tX1kzbh8FuaCLFpSarG12pWDXlqBZeAAOlAh\n3WhYFbHhKyJH0MXErk8mBT2zBqLuMb8TPoCgPpndgmc0xKnR5bl0yTOPiGL+\ngyGalMiTKTnPidriGwnDiSBq5HDJQo/kjWEC20wAWTWRe7dcWhX20B/zOpT2\nu0KA9UyLbBDvX/fZ3xbN9ZuKstp/YSVjUMvNKMwBK6A9SqPhUHaFBr/aFp8q\nIcToQDBnoTtZGDUTWzAVxwiEK8LuQkTgEo2UmgeKthGMcEg0T2tkzbyIJk8S\nEfME\r\n=uUeP\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-dispatcher-4.12.0-beta.20210322.314475a.tgz","oclif":{"bin":"oclif-example","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"tslint -p test -t stylish","doc:readme":"oclif-dev manifest && oclif-dev readme"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.11","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.16.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","@oclif/config":"~1.13.3","readline-sync":"^1.4.10","@oclif/command":"~1.5.19","@microsoft/bf-lu":"4.12.0-rc0"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@oclif/test":"^1.2.5","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@oclif/tslint":"^3.1.1","@oclif/dev-cli":"^1.22.2","@types/argparse":"^1.0.36","@oclif/plugin-help":"^2.2.1"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.12.0-beta.20210322.314475a_1616430990092_0.2574355316028534","host":"s3://npm-registry-packages"}},"4.13.0-rc0":{"name":"@microsoft/bf-dispatcher","version":"4.13.0-rc0","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.13.0-rc0","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"awentzel","email":"awentzel@microsoft.com"},{"name":"eisenbergeffect","email":"rob@bluespire.com"},{"name":"janechu","email":"jachu@microsoft.com"},{"name":"nirice","email":"nirice@microsoft.com"},{"name":"ccf-bot","email":"CCF-Sec@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"d4f68846130f224c6dcb5520e3b601691b100ae4","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.13.0-rc0.tgz","fileCount":587,"integrity":"sha512-kdHPNWUDWpziWGVMYIS+sfBiuNojgmswFOQ4PzIEc4YXbDFJCVNL2GPcND9aD/zGrilUWyufZxWU14Z2E1o2Kg==","signatures":[{"sig":"MEYCIQCwBFGDPgH3yAwvz3yPB/BGKbm6MA7av4HGNw7+pCLnEQIhAJ/9JtCRh0mbYCqFDYhxpeK9i83SG72HdTfF8FC/FMLU","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198639,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgZmpOCRA9TVsSAnZWagAAmpcP/jt3evmIQaHlZEeaNsFx\nkGWT0m6FeNvH9Dzn5ydZD/2ubA9M1khy7WzjuOByZw+9Y3OWRAZlOZ1gvibZ\nC7bY2+YOAjwTXKch+i4btN5+bWUPKzLC4AFhTToqnQXUW992hbGyNDF9+oit\nPbHY1uqnoKES+kgsPuh6ZtQqP8uMsTTicb2arSrzdMD7U83I17s+orFi7hiO\nlEI1gHEpvvRyL6fg/KivTGfaCWuYUZKx/OIacPcKLCjU8vknkJVVTshBr4A6\n3gYkR97g+4ZTJS+RllwinBUg2pn6QWRuBBMcCmpeZ1t7hkggiAGgdX56z54t\n41WcEey7PVb8X+EZbzQo1eTv6Cav7tp2FdKqV554wEmTVWWsB4rFFdKbpkj3\ncd9qVO56BI6xOg2/FAfxitDzuZccPe78EOrZaGDfbz8aiLgOw69kNiy0Am42\n7V6z6FYHF2bEnmKZXg7bSeTLXeKVUqpo7EvcDZkYb69qCeanUi++s0n+5G3a\nvhyGHPikOoXVhioU5nSFZ9WLDHmSXf2VngiFUKY8HqFbJrNYDdMfQvaHXCeB\nHPeVCs5INGmdciV5utWx37WnbTJBWIsyGyF/cM4/ONeycKqHAIMYMlNfiXDm\nGo3UcxQ8qp6rY3FFcDqn9bSH0H2thMJRJ/7KMAkQ/luvuD77Eo6271MMVm3o\nwik4\r\n=7Hy6\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-dispatcher-4.13.0-rc0.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.11","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.16.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.13.0-rc0"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.13.0-rc0_1617324621725_0.03148462386912332","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210402.7e5fa93":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-dev.20210402.7e5fa93","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-dev.20210402.7e5fa93","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"awentzel","email":"awentzel@microsoft.com"},{"name":"eisenbergeffect","email":"rob@bluespire.com"},{"name":"janechu","email":"jachu@microsoft.com"},{"name":"nirice","email":"nirice@microsoft.com"},{"name":"ccf-bot","email":"CCF-Sec@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"eddaaa8b1269ba9b39e8e10c0176602aa46ed484","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-dev.20210402.7e5fa93.tgz","fileCount":587,"integrity":"sha512-41kHtLOj+FnBc0NeKp7iAvHaeA2XrbjbjSmh/cGfb5520/SEjO4xlDk18eSiP+dflgyKYvRQwqzmeZ04/It8Xw==","signatures":[{"sig":"MEYCIQCsc5Y+N2g2g7v81UYImy7GcFzGr6qbncmO7I96OkUlIAIhAI9P9aLeK/4U7zMpNDcAZ0WmA3AS+vbh8e2Bjo6rXpwS","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgZsXqCRA9TVsSAnZWagAAsIcP/1qDC7iOQ4ct0KNKy7xR\nkP5UrdOO751qw1j219f9AFuicg9nnaSWJfFV0z6kx3+UQeE/qTdWlm5fHhn1\nBWeEJnX5kN0S+qwohhXVrvc5hjP+pbmDzMGd+v2jQ9SbbvULga5jV9iyrEoM\nT586uUwR3GE604kulzGIieMbQx2etRz5DdcIF4VvQ4b5BgYcxtFkXm2S8JLW\n+tf5u+0xYP3qRkzsw5ZC0RNkhHL8SI/zOk4kBayUL4otUlWThdgH9wY1a+BE\nIrEMk5zaVRu0iDPs/59hFGunirKctaYo5mfFtUDxuFiezsWqbYpBpXDbzMjq\nqlV65MQdTh4mzRvg3nv44IQnOkgEcaSp218Eq8vOg5AdzblTdZt9Pvkfrt+4\n1HLy4R3W0/9fl7h2yUJKKH7X17NpLDqhET5OMGVMd+XkNi5vNcBizvslcaKs\nwkgHMjGOjkJ/PQb5tyqP4VuQ5dP8kxFqvAyJb5619akw0d59/zKIP32+OQnb\nvLIP3NkzDp5ZJ1pqNgPp8mc/HTxEXlR4xBUqqSe2XDlGGmJIPY3jNX2tJAZK\nTerZcetA21U1LLC982w9ASnNrsRLHu9DFaynkoM1B3JR1sTN0GuwiuN5srLQ\nSyYmrjrcdCunpE7/+BGn/AVCLA8mqMNzcA6G8lPtTXYob/7P/WH1618P5GdA\n88At\r\n=X86f\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.14.0-dev.20210402.7e5fa93.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.11","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.16.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-dev.20210402.7e5fa93"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-dev.20210402.7e5fa93_1617348073821_0.4278626813225672","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210406.74baacc":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-dev.20210406.74baacc","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-dev.20210406.74baacc","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"darinious","email":"dahoove@microsoft.com"},{"name":"benkenawell-ms","email":"bekenawe@microsoft.com"},{"name":"abrilgonzalez","email":"abgonz@microsoft.com"},{"name":"awentzel","email":"awentzel@microsoft.com"},{"name":"eisenbergeffect","email":"rob@bluespire.com"},{"name":"janechu","email":"jachu@microsoft.com"},{"name":"nirice","email":"nirice@microsoft.com"},{"name":"ccf-bot","email":"CCF-Sec@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"5dfbba23cdeb1616f2adb4c11eaf5f7502009ee7","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-dev.20210406.74baacc.tgz","fileCount":587,"integrity":"sha512-vIi1IFd/nSqOqWX4YyNxPLqnzM0XeaKEkPQguZCo51fCGqPVx1mOmExLCbi5LBDcaBz3orwYU05vA1YP0bQFMw==","signatures":[{"sig":"MEQCIBBKC6nQeeOvQWrDf+6vnFALPgL0kXDWz0XfFg9JLud9AiB3GtJMhe+kMGhkm2uJ+BRNRC6V02r0h+itfvK89CV9qg==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgbAuWCRA9TVsSAnZWagAArJQP/15wbV2JXRbwx5Cs9t7n\na5wXh2W8oDmxiTnsEmwEu5Xamu+67daLu3okMekoB6XUf41aexMuowZpCKj2\nOOvhlDTfFFbWd+LEKMYvyOmqV0wbKghrHENxPqxl7eO8gZyqB22K+BqVS40R\nJw9D9uKJBexdidD3vOvx7nL1dFXvyNQ4nz4koI20aDPBvxiVjzcC/IVNYl/l\nLdgZJAXMHTyqwG+DJr73UJsTRU1iE1vUQlYwagNtQkD5N8y6shYvagMx+JDY\nSXRt8jCh7oExsGCrc0CQPlZCmDaODm4Qot1oV+GabgdnGTODklmwIxC99x6R\nj8eOwdSFznvT5r3M/fFGBi7DOixgj4ydkaOJNxkBIzueeHp7t3CwG+m7mi24\nhkp/N2NXvghqYCpLIkm+MIxwJxxGS4ScXaqgp9QTpDnUKjMuqDPy8sx/c5+Z\n1fcAxuWAn1VWdCG8N28uuIRyTHNlaziADlQxotSE9JOrj9gfKsbT5XQ8/OcH\nuQ+mMjUtbcmCC4GuB7534gKHiZxUGGiexbVFvvV1JP5GeIBUnBhb0PsKECcr\nC3hB0U19ILeRX9zKlndiFZwJvRvOKwcC9FvIOX06tKUmN8vrJ8gNzpXV+CRW\nr6CoJUCsVOXNGS5E8U1osz9UznRnCSgoc6qRwpzWtUSrSUzA0kHGa3Qj0Irw\nzytA\r\n=h+Ge\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.14.0-dev.20210406.74baacc.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.11","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.16.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-dev.20210406.74baacc"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-dev.20210406.74baacc_1617693589910_0.6209142520454582","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210407.6745788":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-dev.20210407.6745788","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-dev.20210407.6745788","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"darinious","email":"dahoove@microsoft.com"},{"name":"benkenawell-ms","email":"bekenawe@microsoft.com"},{"name":"abrilgonzalez","email":"abgonz@microsoft.com"},{"name":"awentzel","email":"awentzel@microsoft.com"},{"name":"eisenbergeffect","email":"rob@bluespire.com"},{"name":"janechu","email":"jachu@microsoft.com"},{"name":"nirice","email":"nirice@microsoft.com"},{"name":"ccf-bot","email":"CCF-Sec@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"4f931d559bc4ac76caca71bbf9341fb78a4d22fa","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-dev.20210407.6745788.tgz","fileCount":587,"integrity":"sha512-AT1IDduFSPvDCium6no82tE4sYJhwbLLpcvjHpQ1A55sB+YIDdLAgiFPO8VyA5cJNJza8KpY6gmqlWhTNcA+kA==","signatures":[{"sig":"MEUCIQCcIsGmG5wOr8R4qpUpP8getPmqjNmqz0i/DJ5jTfSmXAIgNqxuXFxys/jbCzPmFNy+xSxObj0IFsohhhnuAOjfohA=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgbkR3CRA9TVsSAnZWagAA/A8P/1+kF5WJUgMhwYCIsXyS\nfrx4ZdgpMQ/y+1YII9v108U57smXBr/7ZBopgOapdIplYc/cl5OJnzyTVX52\noPSG3GfS+CMNwkg6H+MuG/hosA1N7DWzjkfjKSqyCGtq0UldgKQU5Hq+Zhh7\nudEUAXhFwQLVCvBam730s/nAPGeB9sQit4LG0r+drKo5fWXV5yaDOy9b08AO\nTNfcDDlRVbIcwleIrBeynBLoqtVUz2bolF2G+/1hRYmvAfK5ZSj6prVkmgaK\nllgs5rZwqcmbnyyU80fgOewWTBcR5NnpIVfIZX2sdo+Ig89/78kSfdJjxou8\nJVEh0UUFOQBzzHuVJYd7FY+Xfkm1o02NUtl48Pp67s8u1TGCDoV7ZrwtJZJl\n3pSskz0l85RwPSMgSB5j2qWcmkQsdosklVQ222u7LtLxyZmLGaL9PPv+iYOl\nU/geT8iL3gWDj/CIJYuQ13zdneKIqzvBNFtj0kuP2DUZe4StMZJow6baAcPn\ndWCkBR4GgdLNvA3NLeulU8gOueRdZAdxEexzDoQ4zC6FV2Xuu50WUz1BZcj0\nxoJ8IUMxkSwcLmEh+pQyCXn3iLqkeNIoyX48U4JSvQFw8PkMOWEVliWCnNTF\nilDpEGMG/+DROKfeqsY58IfWfPkhJJdzwB6jFNWtEosmpwuZ/wZkMWs5O49W\nV85P\r\n=Eg0h\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.14.0-dev.20210407.6745788.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.11","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.16.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-dev.20210407.6745788"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-dev.20210407.6745788_1617839223232_0.09675062436743675","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210408.393d33d":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-dev.20210408.393d33d","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-dev.20210408.393d33d","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"darinious","email":"dahoove@microsoft.com"},{"name":"benkenawell-ms","email":"bekenawe@microsoft.com"},{"name":"abrilgonzalez","email":"abgonz@microsoft.com"},{"name":"awentzel","email":"awentzel@microsoft.com"},{"name":"eisenbergeffect","email":"rob@bluespire.com"},{"name":"janechu","email":"jachu@microsoft.com"},{"name":"nirice","email":"nirice@microsoft.com"},{"name":"ccf-bot","email":"CCF-Sec@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"0530494fd6a4e38f356209f5ee9cac271b297acf","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-dev.20210408.393d33d.tgz","fileCount":587,"integrity":"sha512-kYxMS97VvYfM/YJlGuj7XB3cQ61PLmoORntu/PGEru5TCKw09SaF6QwO/Zd5Fb1onKJjUjHjtd3trAnGGftOAA==","signatures":[{"sig":"MEYCIQCNylW5GC8dryWkk8qERz2Z6niqDr7nWT21UkKbdVwBKwIhAPIVNsDKDJoH+mg6msJFatiK28HXq+lXq5YmvaVXVkt6","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgbq6eCRA9TVsSAnZWagAAspcP/j5Agtwguw4X9BM8rQZx\nfve8hnryd+egKABYDK8mz0BBpLqYXxayCtCNMPm0WrB5SHdyAxy8sxoOr2f2\nQucFxJCc5l2krfhcQUyLxc9VY2EnZ/A5ucAG6ApCqN4rGXl9MwfWzZd4LEXF\n5h+S1yVRghM44/nCvLXV62NcCkHRUgT1lHkZF9/ktNwupA1eVpI9HJ2j3yDD\nhwQkNvBT2V1O00Mqi4Cj2nAtJ+8f0wFqzB4LTink9BdzYET6O7FTr0MT4dxv\nTKzoaKfM0IVnz5Sv11q/h+8Ia6E5Rzh+uREHO623a5vf2gQnddTScI/AJdHQ\nmyDyg+26hujTOueatjynvGJ4VD7KydEfjlYF4CzU8ChY8VCzUB+C8aUXO3oU\n9R1r87lHylZ44fct+WKKcY3wirxou14BeVYWbUO6Wae7mHlY4h0IkuyC3Wxv\niW+Ljpy3tdETiSKyQ5PP6J4RJE6BA02CxOJoXsmFpZ8Uo/2Vhl4rAyKYBGhm\njp7+4JwiVe4ANpNEBztFJXHMrZ4D9AGj7r5u+Uq21/XtQPRPeRY3I3CkEhYm\nf5u3JDt8I5/sjhEzr+vL0udNORsRWKTtixIdQ72D4378ZtFGsFP4A2P8As45\nkBsQ2NuCPD+XZTPRI/XlFRqE3fOcwKdff4I97DNjARxhpB0aBbuciULrRnAT\nN27l\r\n=+u+9\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.14.0-dev.20210408.393d33d.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.11","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.16.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-dev.20210408.393d33d"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-dev.20210408.393d33d_1617866398226_0.6720736075158971","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210408.1a5070e":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-dev.20210408.1a5070e","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-dev.20210408.1a5070e","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"darinious","email":"dahoove@microsoft.com"},{"name":"benkenawell-ms","email":"bekenawe@microsoft.com"},{"name":"abrilgonzalez","email":"abgonz@microsoft.com"},{"name":"awentzel","email":"awentzel@microsoft.com"},{"name":"eisenbergeffect","email":"rob@bluespire.com"},{"name":"janechu","email":"jachu@microsoft.com"},{"name":"nirice","email":"nirice@microsoft.com"},{"name":"ccf-bot","email":"CCF-Sec@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"21f095f3ff1ea89a8323e52fe111281fbfd6d56e","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-dev.20210408.1a5070e.tgz","fileCount":587,"integrity":"sha512-r+ljXRg3jP5EUUurFe7HeDkt7BbmAeUP2mOieJiz45rtR1nRS/UxtIjDilyuUxzF+w2VM76KbZdYHvP/xaptYg==","signatures":[{"sig":"MEQCIChDqTed9Djq3fyCBsy1bdf/KKhGlyNiNRrYHcHIED0AAiBDwGQqA2JINsnXxtTUiY06NMDjhBx8hqlEnfnP3h9jxg==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgbza0CRA9TVsSAnZWagAA0YsP/07SkOBHP1/z4vIpmFQ4\nSlMs5f4mmVG29hSZEpOBBqp/RpsIviQA5FxmGuBTZH7JU/OtS19B6oYZxmzL\n3KwpFgdG0txZRmzDDdOkFTwsjqvPUI2QJgTPA53HA6IUZOAHKeriPlEiQPZt\nfNzGWls6DUWdWD9eu0sWG2ftTXEPd+s/s4L/aQPZToqRMIBcXfQqm5BPYO/G\nzEzhJSFbh+teSTdZbWg5Utyaa7vVSyfg21mmtc/Y5Buo5l2i4BedG2XGO2+K\n26Rod8DKKQA75+Epme7JD9qXASBajJDLFKivCkVoJh7uxs9uAUDPwIsN9JHZ\namF5mwd55aXHg04DFPkZiLvSvX+hVahNP4ci/WxhPu1Lq1wZza5Frq7BDNSE\nG+IJOcIgtSBtlCxwY9K/KowGVKIATp9P4YEqYraHZcHfxfk4O58hD0CutZUj\nBKMCQ5je4mluPw/mP937wHdpgps+D9ecE2GxM3W6G2+QObuKugffbUvT4WiV\n0cMCwy0HIgm8Ljlfv6tbwjseOHr7tiJb3SDP3YAjHiz2f/eFZUUR5x93epOa\nEX3CtcPu/4xe2CDLIvtDeFosmMBFSx5aU6QbzQdcshqhIda/1oA4k4EFfZHz\nrE3QeiGgsqIWlSEFxqqX53DXxWYuLeacvCfjiKYybKbYcVi0vcmdNs1HUtki\npizd\r\n=7qtx\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.14.0-dev.20210408.1a5070e.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.11","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.16.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-dev.20210408.1a5070e"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-dev.20210408.1a5070e_1617901236506_0.3299622108454403","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210409.1a5070e":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-dev.20210409.1a5070e","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-dev.20210409.1a5070e","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft-graph-toolkit","email":"mgt-npm@microsoft.com"},{"name":"darinious","email":"dahoove@microsoft.com"},{"name":"benkenawell-ms","email":"bekenawe@microsoft.com"},{"name":"abrilgonzalez","email":"abgonz@microsoft.com"},{"name":"awentzel","email":"awentzel@microsoft.com"},{"name":"eisenbergeffect","email":"rob@bluespire.com"},{"name":"janechu","email":"jachu@microsoft.com"},{"name":"nirice","email":"nirice@microsoft.com"},{"name":"ccf-bot","email":"CCF-Sec@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"ec1332bb2f499c719098223665b191b0b088673f","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-dev.20210409.1a5070e.tgz","fileCount":587,"integrity":"sha512-Cg1zjyyHpwenWw4NC0JYmExDavKA5/mI2BH1tUZNO1Ay37copHwI0BIqcg48uoD2oQMbZ5NoKjEtMMDdGNVTyQ==","signatures":[{"sig":"MEQCIDJ8vROJsdtmaRF6jMY3T2LKFywq/q+kCDle06pKnqttAiBVPgkKDXOEfF5+cUKPzSyCyyD+i0B+FCo2vvs1Il/U8w==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgcAA4CRA9TVsSAnZWagAA66YP/29bh4WOnM4SwZ/t+2pH\niZD5bRGjxC1wtyAFA3ACawxhjLb+h0IWosnPFeFxmzezlzlZDfzi5BQG18gu\n2Q+KOJgx5DghnflHTrDWmQnjwAc7y1zBBB2SlrlpstMNYP0VoxXw2EEfRX4t\nNpqjHsXZ4BZg/4loYiBQLNVeKCRknQEf0wHXMvgBn96HOZIAOMJ68RX2sfEX\nf9k6h9DbIJ8pXctHMh0Z3Ql4WO5ogAXrGxSEMpx5MMYCyCrNH/oLmO0Iw90/\nKFQ4FIqf54xq4s/l0pElCZicCIBBgLMe/V7JiTheAnbC/APYmiwfLuFfmAvk\nGzoLk5krxYegLTxNCWSzNmqAMcJK65e1Z3YaVT95HXUJR7DmO65tvIWRSxmR\n6QIXyKBqd8gCWd8zhF2VnOUy6/CJoEUTD+DmoJOCy/UIkQqjJLB/hR5TsgR0\nC9n21XgdWXpD7+4IA78nJWajdanRw9AQs2vMcTYfI4gjkR7Hizs68diXYJa7\n+FEktYjl7JGPHjHPtD3V0tJKN9Ka8EdQ7yO3oQbIzOzRG9Bj0foITQLeixgS\nzM1Pa5cO9FE4bFvmoSXSEp1yRwP6Zmp3BSWu2a9Apt4NCLma2M7JOgAMaO4R\n+ryXqJ4WuZpTsWFi4wET/fKs+hukDByB+4AeZY8j1+tEMDv8pgXrj6cBt4Lk\n606m\r\n=XhQX\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.14.0-dev.20210409.1a5070e.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.11","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.16.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-dev.20210409.1a5070e"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-dev.20210409.1a5070e_1617952823779_0.24207208924238244","host":"s3://npm-registry-packages"}},"4.13.0-rc1":{"name":"@microsoft/bf-dispatcher","version":"4.13.0-rc1","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.13.0-rc1","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft-graph-toolkit","email":"mgt-npm@microsoft.com"},{"name":"darinious","email":"dahoove@microsoft.com"},{"name":"benkenawell-ms","email":"bekenawe@microsoft.com"},{"name":"abrilgonzalez","email":"abgonz@microsoft.com"},{"name":"awentzel","email":"awentzel@microsoft.com"},{"name":"eisenbergeffect","email":"rob@bluespire.com"},{"name":"janechu","email":"jachu@microsoft.com"},{"name":"nirice","email":"nirice@microsoft.com"},{"name":"ccf-bot","email":"CCF-Sec@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"cf063ca4a3782a303923092b45d00bb27562b6a8","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.13.0-rc1.tgz","fileCount":587,"integrity":"sha512-rRDkL7Z6gqi7RnwhnXL6BaczF4G/hE8RNCsTC41pal8Q4cgoSoi+rJjbWF1ZGwanN6QFU4ai5S4OnqGBQ+ef1g==","signatures":[{"sig":"MEUCIGjgqPyxmkfCRqlqqyvh18g/INGsO0/kAqsdwpaTAYEXAiEAqvKiKdSKmGhtTT/lhJe7W4og3kv5VmLGTQu//UfF7k0=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198639,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgcI8sCRA9TVsSAnZWagAARTYP/0duy71XWOjjyp9bRwDS\nOr+/xL9qHEC3DZT239l8FRMtxVLL7kPM8OdWzuD3Ig3fjFoh+9Mess6E0qOp\nnxnNcZPcSingqa9Dg0G1E50zamszL2hlBAdw21s5KVDwb6UIDeUU4BdvyuaC\nQfxpwiH63M0KjKALGZegbI1pVHrJps4I/qPiclYoRCD9vkQFityVBXGRbQL3\n9Tll8op590+N0z7NwBlbvsHW1gFMFYW4puUcXFY2SeKuYLqzdtqX9og7IhkD\nluxkbkCfQN+h1wdLd//X8PIFSNJyrHLc93HQdZ98J+cL4MvIxmQHqWTGs117\nBdpUEmNpCMQFa9rvmw80D5GyKV90MZudbRpd3QCN0f2lAxJAtU93WgH9pcrk\nCxaI4I8R8yb+fKXntDkD65ojRn2pUGTmL2R/+f2+0QnfExVQF1248Wvbruim\nVq+wvQ0gsh7xntH5hz6epuiFToiW8uH6F8quzEZQynsUvUHSQ+hLAjqsSZAV\nQo9FCoDaZd4OCanPNxwWaT2y02F1JAXIjePKViOH6/6qDYrlFpLmHIVSSyub\n/OnRKTSkTMTl8yp3ecxn0ZDd87DwzVDLSwnlFv5YnOTUBsruEtLFOv0I/oVn\ndVYyJHlNzJr1ue76C80Qq50YRsqvseGNl4Pt3CkAzxMUjLoqM0YUHyYyvcLC\nvsbe\r\n=mT6Q\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-dispatcher-4.13.0-rc1.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.11","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.16.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.13.0-rc1"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.13.0-rc1_1617989420145_0.9653841658444868","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210415.161c029":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-dev.20210415.161c029","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-dev.20210415.161c029","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft-graph-toolkit","email":"mgt-npm@microsoft.com"},{"name":"darinious","email":"dahoove@microsoft.com"},{"name":"benkenawell-ms","email":"bekenawe@microsoft.com"},{"name":"abrilgonzalez","email":"abgonz@microsoft.com"},{"name":"awentzel","email":"awentzel@microsoft.com"},{"name":"eisenbergeffect","email":"rob@bluespire.com"},{"name":"janechu","email":"jachu@microsoft.com"},{"name":"nirice","email":"nirice@microsoft.com"},{"name":"ccf-bot","email":"CCF-Sec@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"ca15ea4a20ff047b1d774f6cd6a8ba7e625cf680","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-dev.20210415.161c029.tgz","fileCount":587,"integrity":"sha512-b6v2z/Djf1RRGadjVAzTgyOmIJ+fPm8F8TZUs7gRA/RRBNLm9awEyug9y0CprHV2azhoilXzXmTtoYxr4N2ztA==","signatures":[{"sig":"MEYCIQCyZb9LxFjRnTjOAYt6l+k8KXFfF91ouSvJNujcG3LIiwIhALvVGOxoVllkgDz5EdhaQJuGkH3h7kwa9xbOU5JYwhOa","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgd+knCRA9TVsSAnZWagAAraAP/jD8IzHV/U+inBDJdg7X\nxOROWRr8Nsfne++L6qqlT5q43YvQWMTiOj3g6n3kFvwXbCDmiMtv8NC3Quvr\ntzAkKEClu8JhCgm1haYZWUiJy085GD5xyD93f7srB6VbIC2sLVjsyMEmko2U\nwweyNuKKPbe71nCEMPJkZmYLh0Xd5mByEDL8Bd+Ln9KNb0RjuU/l9yq2jd7k\nukFATYI0PliJfoL3NHjJAJimCouyEfgO24QqDS/3/2msNdhLKTZqWbdzWSWl\ntF9dEmHKG8LrqMrb+taTilSJv8ZKjSw7DD6QoR11as3Tlun/0dcJwQomeeMm\nvr8hMKOBcGq1rkQLRh0dhPD9Jn8H2+Kjqr/4xmv7O7EMciaFKZVV75DcZToJ\nrCRJ0tFcPSK9SrCZ16/14h1g8OwCEyPeO1kXVU/CG+NQeJ5bJmicRNayO3gl\nyPPvOhtZVpTM/wZXRrdf7vHnd8qDVnA8Xw9CDN8NPv4uxcLDqKXWQY549PUI\nAFUPdWHDAnnvvySNcAYBVJN90rF4a7nq9xpzsKTFD/D7nncDm0zj7asd+pF1\n5/MxwASkvpTSJVQHPe5LVnsfpTF8q/oGmqkMNYLxaYrdXvuXuvmtGghJwUTI\nMsbtcVzBykz72or/TfDHhef7t3JmqIizeRe18n5S4YV9nF7Dh0s628Vwr7ki\n3cp/\r\n=UUlX\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.14.0-dev.20210415.161c029.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.12","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-dev.20210415.161c029"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-dev.20210415.161c029_1618471206877_0.49135311657302805","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210415.7e35288":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-dev.20210415.7e35288","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-dev.20210415.7e35288","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft-graph-toolkit","email":"mgt-npm@microsoft.com"},{"name":"darinious","email":"dahoove@microsoft.com"},{"name":"benkenawell-ms","email":"bekenawe@microsoft.com"},{"name":"abrilgonzalez","email":"abgonz@microsoft.com"},{"name":"awentzel","email":"awentzel@microsoft.com"},{"name":"eisenbergeffect","email":"rob@bluespire.com"},{"name":"janechu","email":"jachu@microsoft.com"},{"name":"nirice","email":"nirice@microsoft.com"},{"name":"ccf-bot","email":"CCF-Sec@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"f74732433e69f5d982fa8c40fe53eefd1265408a","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-dev.20210415.7e35288.tgz","fileCount":587,"integrity":"sha512-egJz7TDPGZc26tbZgwfzzPGdMEP12cH/T4bmBuPss005QjeMKnBinGFC0+Gf/YX4R/O+io/VwlfltyCRtAx61A==","signatures":[{"sig":"MEUCID1lJIMvbKI7/QB9NRBexq/cV9BVjVgNHBJIsVK7VGSNAiEAyX3WOpmUXCc3/8gvyAYVuRxVVdistbv7d6xPuSqjks8=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgeJCsCRA9TVsSAnZWagAAwYgP/3YV98Sh0RjtUKUrQmiw\nze6u6V3VbzzDtmgUBd0lw7oFSbmIASzahpAAOJlPMdFm00+lphwaQ3PYGsZw\nL5kVQ136OSwtAKGY74ECf4dfA/noGbjwFkvLXwbBYgGENnies5KQM5UXDrpk\nTvSG5S1drvlpTCdVYubnEvcmN7U7/NZWigxr/Oyd5nwUvNgLRW6jWfkstyDf\n/5sQTYyeSNDJpLZBX9jdDLGSqP8ahd0g2MdHqUUeJQg/fuHCPoms4am4QuLg\nVOw3LdIcfC0fk1tDvDyKW+1/nXsO4FI5yh+lDt8OA4PvrfxR5Uqk47jh6pbi\nfX/FpMDshmtQFMfgxIB/BRxPAQNg7t5H78mDYrNIhg6IcDq1O2B5+2UMOURg\n+iKF8Hb8oTCFbrm8kjS/nFGilBR8iG0BTZZcLyR6Y+yl8p4RTvG2G6ixybmM\n2+z3Z+LRxDv+QZPPm7ygMD8R+MgqjyXd3+MrliRZi2J92YFA8fu1XARpwgaG\nmKv/jCV01VnSqOnqI+J3duKWUpIymCZhy/dCiHnbdc1GuDawTqrIkAFB487S\nz+GPIfpwqPVQnLu7qPySoPTmBHLd0YcbZd5DaDUo+gKM3wWe8TdFaB9jBR0w\nTXrI7+fUiPcMSYst9F/AZtDb5ftiqVfueXY9BBVx3RkdEy1h08HZNuJTnqfN\nLtfq\r\n=B6x0\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.14.0-dev.20210415.7e35288.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.12","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-dev.20210415.7e35288"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-dev.20210415.7e35288_1618514092062_0.08154981777839954","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210415.ab0fc09":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-dev.20210415.ab0fc09","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-dev.20210415.ab0fc09","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft-graph-toolkit","email":"mgt-npm@microsoft.com"},{"name":"darinious","email":"dahoove@microsoft.com"},{"name":"benkenawell-ms","email":"bekenawe@microsoft.com"},{"name":"abrilgonzalez","email":"abgonz@microsoft.com"},{"name":"awentzel","email":"awentzel@microsoft.com"},{"name":"eisenbergeffect","email":"rob@bluespire.com"},{"name":"janechu","email":"jachu@microsoft.com"},{"name":"nirice","email":"nirice@microsoft.com"},{"name":"ccf-bot","email":"CCF-Sec@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"ab5c7ebbc4dc451353817ccb01ada9971da28e75","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-dev.20210415.ab0fc09.tgz","fileCount":587,"integrity":"sha512-LwSNs9bL3y7EWLvVpzkoJgvW4eqGQbFn044W4yFv6poU35NC/+ChxKHHBJlxRukUt78mIbmmjM5SizFOKijcQw==","signatures":[{"sig":"MEQCIGnbYJZn9jvv1EEeXRAdTDThWAU0P+o46HfRmip8RMdOAiBu9fMd7WxdGv6zA2MFxNFDpzsADWi55RqXMib28REvmw==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgeKCvCRA9TVsSAnZWagAA4LcP/3ZV/ItdaWHVCK2Q5qIu\nHmk1F44J5oJG5vrTg3u1Gp17dufxop05yMUdCFbn257GCd3Cp04fqfPMyozE\npMzIE6QGyC37yak6TZRLhsvs+au3xZfIOXvJVnLrC0CYVPA/0RlcA3tdyfry\nFqn8Ttv0z6iw7PqumDo1/apc/3sO1p4AcaomikpTzKB0FNE8c0joSBzf7Zy8\nMKTIwB1xQ7UMDwmENQrleezc7vDbFRnudd4jYLXO0dw2Aa/3utdzgAjj7O2N\n3cB8Np9th8dI3GIpWkbsoK+HMkrifnI8sodFu4WoDJZxrnwMkinZlt/sT3Y9\nzWcS35h5gbcvOl91ahcxCeOND2AY6dFPgSAxp9X60GAL7j1HrRBQxYIyCEA9\nxkn5ISt6DoRd+a3TRu/PFyjPJ1csHjyKIF74ZFu/OC+9XesH7P8EgIpXTTVW\nHn/hz8Ff878e2fNZKE7coK/kgkYSIrTIoFsNZOlH5mUID5JqZc7LZCEeLB/X\nhBb6uGYiLXQ4wAKKIwx8ADioWEQoCf1fvXC7y8yl2q8QSZ1lOiGgQTLLCV28\nGAQJoMGAkBw+S6BtfI4zhrlzdp5t0f5KlvdF6qG1XeBSDsuqLZVo3QIyxWrM\nkh5/ZPQ9owN/ibYg96uKcy2KXQzcQbyEGBtBG4Xrj8qHs9V55FX1FDD+dkrN\nDuMB\r\n=ibK6\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.14.0-dev.20210415.ab0fc09.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.12","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-dev.20210415.ab0fc09"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-dev.20210415.ab0fc09_1618518190914_0.6547427335685665","host":"s3://npm-registry-packages"}},"4.13.0-rc2":{"name":"@microsoft/bf-dispatcher","version":"4.13.0-rc2","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.13.0-rc2","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"490ff95773c9d3f66479f4fe86f669a9409fc002","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.13.0-rc2.tgz","fileCount":587,"integrity":"sha512-9AvBd6Mqybw4fLZCamHER+9gnoLMTvpSmFQCv59FOSJiEjt/KHC+yYGAh9g6qsxIAIovZsmssFUiHYgdypFvNg==","signatures":[{"sig":"MEYCIQC72wiFWKdtWtTZXZkhqc17Km4a5Ia3w3xtmYy7XWyPvwIhAIASyZBI03aI7KuPWbMDeYOhrHNhJ65Uo3tAsI91XbZA","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198639,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgeMMtCRA9TVsSAnZWagAAaqUP/3JMypJDkBWDvw0RKeFs\nevMslepbZoBLTwHQFJXFGQuHqlQy7JbPHl5dRf7nRPM+2J3iJzDs0Bg+g7wM\n/eyll5bAp9tPa04FGbd+tkClT/8Gc/wzZcjSSUDFt6lBsLS0c7e+thfGMCgD\nPxcMQLvLnwTSuv4wdPoU8Ln5NqVFKufa1u/4rZq+IFkXCu4W+6Srn2Zw+6LR\ngkbKW546GmQEiEwgTBJKep4GaZ74wj7y39haWCMFYEci0HzTpmdcaZesGgii\nGKYESQ/nJY8Th3Xe4OSIZtJXwosocbcxGn3Oim+g097uii4cQQhOtaQSDr/p\nntezwyKb4ZEmZHxCk/QcvBgOYLtNS4e8IE9Jzp89AYTHsEynbmJ9nV3MIGO4\nZhzYy64QfzRswdVtl9T+sHuvj8dWS5hW/zAnOWRBFJgo92N4XqGeJmts2Gsx\nckeyujubrsIOWKvXJtFAb1/KmCqBgQpP5v1yU0XNTXeTddOuhq2w2A6X55VL\n21tiT2aKa3eLTRIMajdz/wDMT7l4jCB8KqguBhzCl0zeRT6IEynH0PvjZ0rZ\nbWmQzBNQxrvOZArYgwtSSDusxlIb+PsyNc40yQGCvwSIFyu2iHZLtCAnszBT\ntmGYb4ISY1zSgmVTWfXeRV+umPTqHUW3DQ7hqTD1XdFkYSuTTHabUEK3XBdW\nWv/R\r\n=MKxi\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-dispatcher-4.13.0-rc2.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.12","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.13.0-rc2"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.13.0-rc2_1618527020820_0.917072454609821","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210416.ab0fc09":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-dev.20210416.ab0fc09","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-dev.20210416.ab0fc09","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"7b2e7bae977cce8d457338c816a0ffa9529eaaf9","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-dev.20210416.ab0fc09.tgz","fileCount":587,"integrity":"sha512-AESfAxYP/7q2tXO2ZW2tFem9d34xPndr0xDB49oM00LxJpF6YfA3ylqzr3S5i2SEJlZAnbXn/2ByInv6LJ5VWQ==","signatures":[{"sig":"MEQCIEzNGyvdsaLqYd9ALG8FVPbY1JjvjxHME/pVtdJ57JRgAiAOs306k1jm9uQVYLHznzheE+E5N1lO7xFy9BcBRGR65A==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgeTqqCRA9TVsSAnZWagAAdr8P/1N20LkMyGX5tEHmd0Te\nE+sNCdPbA+6WAlpar286mAPOMkTZneW7P/YDH7TmeWkBvelVDdNdcqSmBRL0\ndIXbMpzeZerHwSio+YkVl9rRrqH1prxu774rHrpX/H9/jeGORPiY7lHb87FU\nD2YKj+aAyJP8R55u/DZTaoTkrtL7mTV3ONj36JDvOK7H6lyTif5CoKmQJoJR\nq2mauzx0xnNFL+4b3d6PVrkXJdmguqIukM/Ll10YVymoXj98ofu1RI0QrNoS\n9QNf1Q0mHxufHve7j5oIp8yBBfDu+TS1cFYibYbNs5DdsRHvijouu84iWTRJ\nI6MO/m2DQKzrESInoAzPy7O2IwoGAIOs/Y2W+J4gCI0JC6Sqefsolk53NYzf\nYeQAprSW63DWCghMFbcZj7rheSj/h6iAIpmKzkrgYK/3pTLYKZMIM/wUu2GY\nH63QHSvdz0blg1wYCG17DkPSJxm2b4sBy2vrkAsvJjpWUDmhOw2bHU35KN0A\nn7J8gytI8OFsvzxKoK6gqxiGxY+9jbOjc9zusNvyVavI6NUym8qTVFrRHeQz\nWQccGB08t66pbcUUBlHiNZUdnPI6C562FrZQZScVB0A1XhgKUdtGD5xgD7ga\n0qX9WSmwsvnI9oh8o4PWLlZp1Vrc0AouSIIXvzi7zLElByTIIrBCv6hfgN9O\nyMFx\r\n=LEo0\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.14.0-dev.20210416.ab0fc09.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.12","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-dev.20210416.ab0fc09"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-dev.20210416.ab0fc09_1618557609680_0.8256523729135679","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210419.7257bc2":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-dev.20210419.7257bc2","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-dev.20210419.7257bc2","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"9786a13c62156814d68cefdd446ae5680097fcaa","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-dev.20210419.7257bc2.tgz","fileCount":587,"integrity":"sha512-A7NkAAttJtBZvt4uM5QD9yO11PuiRVEA6veEOtOA+BeOAogV2N0K5/gw0b6SSNSWLf47RbzJllTzh5UH3cupsg==","signatures":[{"sig":"MEQCIEhlfp/m4f993NkBRz3ZcWPy3Mda8LJJQJMXDHVBiHSBAiBelQxr1wxSJrXwAOjRN0A+M7m+G17VAJSSdL/nPhpUDw==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgfc03CRA9TVsSAnZWagAAudYP/2cV0oxXLMy9l7lMheS7\nII/XZyToFhFbthwf+F7Nhb3cgNCtBLoInKtpBvmHJ98MNJBoqdePea2r9Caz\nOhWKWoMpTRoddS3M2hFn6vwzAqUc0K792xo0Qf1mI8ee6Gnsgqbg9Ptk2lqw\nczkfiAT7uclTYYxQb5vWT6Ebx8X0UZ70v7Po5e0rK1YH/Mg4QvQua1RvifXL\nZiozCM3/aOj5CEjOUS9VNtv4J/coy2GdqQkNxX5LCYf7eZpXuB0bXDppQBFo\nXnUzHtx+dh7acJTVcmLgYf3wWn7Oyzn7V+ZgqluLhyY/kdRIeVR/ctuE9HqX\n/TR2XSd+/iXucmd+5Hbwl+SNXwgDgt/RrQg5jykHS9bWR+R0oR25JgluFKxo\n++ToQRIOKsVCkn2gHHUq5HE28MCous+JacDgs4esdGxrarvkVQiPovAUs8EB\noj2ekRV66oHCAM8I4JaVng65C07txrDEbLz0WXdJl3xSuwlMbu2ClIvWrQx+\n9menvfxt44B6VRFXdUbg0CT0CsPrHFTKiSDisNHKajPTyjmv+X3mUjWyCdda\nRJxZ6P2noSe7A7vXHcEHf7mYjI9P2xwBSs2EgBTRIXlCSNK2ZNTD/CUDdIvj\nAYOh+SkYY2bZHPgXhGdqshpF6giVrWv9eWKChXdvh0sn9+AoiBKTMOcustXz\nNw+U\r\n=Lzj1\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.14.0-dev.20210419.7257bc2.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.12","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-dev.20210419.7257bc2"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-dev.20210419.7257bc2_1618857270511_0.19478651511619227","host":"s3://npm-registry-packages"}},"4.13.0":{"name":"@microsoft/bf-dispatcher","version":"4.13.0","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.13.0","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"63a89d5d7a43ab342a5bf6c54d8162cd21194798","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.13.0.tgz","fileCount":587,"integrity":"sha512-Vv+LeQEMs2gd/hh6gzkr9DnHE4ShQfCjdEDDYfwFVpoyzSDqLsjakplr3SgCx3roCeJgkDwroexx1lg1gDYbsg==","signatures":[{"sig":"MEUCIDgkkJTaKb4NssS8DwI4FxGcxHA7nM9eK2HK2mDY+6L9AiEAo7nQ+TnDoHxqoMbr77mNQm9AU3R7fMpQjear7Bk+x3w=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198631,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgfgHJCRA9TVsSAnZWagAAKugP/3hUWWmqwMsSJXVvBYwK\nMr0IeWxX1pT77OVBZyiey7AJBpiNr062DrIr3QnVy+zrTtECkOAw4FTP9kHq\nB65QzJckMKd14vlTiD7S/JneujL4aKp63U8E1uVKvqshGWLKCi6RVjt0gBQE\nQD/oo+aCSHpKPYBtHiCzPgW2VKk2kKzd7+9fkEIqGNjyLS+jgt6hWBeZIUDb\ndu7ULvN4FoBqi0y1a7ciX9uYDNiCZdpxdWh9vcYThEqoh2TWmIS2wT6e5ygb\nhLWzfg0brLH3QY8Un1LDzBvHQ1VFq5pd1f82ge7zWmwtTT5maA3tt4OI2H99\nOKOu9I86h6RyzBLTipXmZkJbd2T1iTNwcoVeK6onSpSS10JYVJqZ5CImMr4k\nlIvTmkBxai+2gfr14rbDrHvtotdjUtI4If/jFt8OIIWMiAMxM6q+Anupx09E\n/mcu4gkIocI1fo0BBAogMnJkjKEzzLh9y1yr47A9jZLMZzOld2BlK4HV2mZ6\nfyKmvw69l8BlAlZl/iSG1JcW5YAOxJ3XPO1Ts6+ETx0fpIMJfQgWXVK2Xku5\nxNfcwT7hKo9UhfiH/13YV+Gl6e2WEtwjPoHWb+QkuaRUtnEZZJtDlAIxU78u\nxGiDNBfplWPCxDlWokz2su4y3/kuNolH12JWqQ+5l/PH5bwmqm3pRGz/EZ3g\nEtou\r\n=jlYa\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-dispatcher-4.13.0.tgz","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.12","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.13.0"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.13.0_1618870728509_0.8104366073756066","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210420.fc92e9b":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-dev.20210420.fc92e9b","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-dev.20210420.fc92e9b","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"zhyuer","email":"zhany@microsoft.com"},{"name":"nintan","email":"nintan@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"14ea6e18501e61ae9c761cb5982e39ff50472ad6","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-dev.20210420.fc92e9b.tgz","fileCount":587,"integrity":"sha512-FHNSrcMvvtI7j4oQikS/2KTGOtWCYCvbxyRJd8uIFsmop99CbVc0RefbdCKDYXOnDOAcRsds4MHrwh1Yy20znQ==","signatures":[{"sig":"MEYCIQCgX7NXjhIleNAaYhqlL9uKTVrnj9eN8hk6QkrMk020SwIhAMSovjofvETPC6SeZjAqieqJuEqIftUjQ7Ju+iiJBxLv","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgfoCkCRA9TVsSAnZWagAArnkQAI2dFq5nneOkDUc8mUsD\nZAxD9tAmjoKhDEVBpQzIL5524RbUMnrJ/6SxLBSJRA2IQsrYVX2++xnX9Ea4\nG4jBhwP9C+YLz4n+Cy3xh4OaqCXn3BlVfvwyRkqCYdFsmeTu/kvlW/hBU7iR\nHX4kbHRiEMqOoVeM/YdN56YUh/8k6rvHQqRbNlADeWSiJzs1LoivBCTQGVk1\nF7LzwX64q4/NPj3oL2Jz9GmSA3Q07fUi20rq3gPEVJvwBO2JV64NU44bwTac\nvV1Z4uicsLlo9Hkusw0xWbdFKNrvRKU1CRaCRX/080qBmtW7gISfd9GZTqCV\nvY2jUki+9fXVndTMgp8zsBkAKjFiATdlrgPRPdQES9jgxxlvHXAdFosS3N/o\nkXH9xOu5wvjARPfUw+Vei/+mLmDaI9VwSfhFj5sUCxwEL4OHdmHB26HVL7Wy\ntjmr47PN7ajnBtKaxL+WIoxI85wW6OKEXrh1UFQ9OdNtGqfZeIS2okRtNBxE\nCrS6jIBz5qWAEWEMGZtRRSoowT/XE9wgdGUlBw2mP8k+j71u1VZjHzdOOHyQ\nc7VJzFyAqelXTSLvbkVQRH7eB+dHwFtMSy7ctZ+ZoGEzHdrEONJjrF/5modR\ncaYmhpAu4crBikqWsrGevwxBLrsgW59/NzgDvhjcR2UyrDVrHo9Y41menrwZ\nIg6m\r\n=eHqe\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.14.0-dev.20210420.fc92e9b.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.12","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-dev.20210420.fc92e9b"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-dev.20210420.fc92e9b_1618903203341_0.5065494864615794","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210422.eccdf02":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-dev.20210422.eccdf02","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-dev.20210422.eccdf02","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"09ff66a7437fc89392a57e2190099e9b87366e11","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-dev.20210422.eccdf02.tgz","fileCount":587,"integrity":"sha512-OGuN3YOm2zK9QpEt/3c6cKJFkbgcCRfzpUHRSZA4my9kktVReUsgFJDo75K7Pst7+DjMcstdTmQrKfraLuIljw==","signatures":[{"sig":"MEQCIBRoXUQSL/lH/RwOe70W66X5YFPnASTScnZuFr2Vj5GCAiAPK2quJ62dQIQ1VFewH1aNfoaDiHDYIEodhPjpn5ilgQ==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJggSP9CRA9TVsSAnZWagAA7WQP/0yGQRmvRhGjCuBdq05k\ngVXw/To2WjZb7WKCJrEHadfKVIAVqelz3g5KvDAQ18EYRHV0ZpA7Sg/olPbf\nJv6qgtwGtAQlPRfT28MC3DJ+llrKNx288vTa9F8JVRiVB/NuPQ89wr69oSQ1\nqA6q372WjOb6tUUdpgC4GbPdAuNO7jyNFRI+RV8cjvILq9lYX5Gzi/LFNRNl\nx5bsCnURP0Wzc1P0vvGZehzRprryXMO22yDbao/u6pJpZO9pr4tMI0JZbFu4\nxhO+9v4dLvYAf44W0inyLkPYZNLoxkJiZtalff7+RXfHWitilMebDnRZxJHb\n4vugX3upUBSH9pL9r8jG+cGXY/GIprGE/5c2aEI6av+Y+gFkkA8kXfhk39L3\n5WfsyKeE6fR8Kq5YJe9E0cjqCnbhcPUuUe/WP7+JakTSQt+pAih18H4C6gZC\nA86ibRqTtRQ6hTQi3AmSBzsZrQL71R0ByI86qc5Wkohsabqg2b22E2P8mapj\nGwEET6irnxliDJxE1opGuBnCSQqvPLAhSRpO1WQ/oSQHszZyAHR6lxFn6qiJ\nAfNfryZUZWoj89KNMDMUxg2yZ3E6x95oW9Px3yu2qxwdtqAjd6Nfnaqh3AYP\nK5kp3/oZa6Q9CDhwd6xzz9p0CmeW95PD6BDj23v73jwxMm5cuYRkSJ06TwAz\nIIIL\r\n=kWiB\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.14.0-dev.20210422.eccdf02.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.12","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-dev.20210422.eccdf02"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-dev.20210422.eccdf02_1619076092884_0.9688935000438272","host":"s3://npm-registry-packages"}},"4.13.1":{"name":"@microsoft/bf-dispatcher","version":"4.13.1","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.13.1","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"1b96f23976a3dd0bd4998ec3655cc54ef885aeb9","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.13.1.tgz","fileCount":587,"integrity":"sha512-7gYyl3h5o+fUo+S4NJoF6SyiDN8Zw9jEqeQ5w1U4x5WRE5F3mSU4TG59A+rLXXgl5SDGq5cgj9Z2JnOUilsF+w==","signatures":[{"sig":"MEYCIQCApExcT6y0dpToKUq+p4utFfLsDy/kQns+Uw5kvGg3WgIhAM7zqsVE0MxbKWBT1zzi330eeogYdxveQn+KwGrQzsGJ","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198631,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJghzNWCRA9TVsSAnZWagAAHkoQAIJoaBRoyXBWzOypfHrg\nntgVdYU33cKaJJm+LDUoS1rQgW6ODaceYcD5VjyDHVG0zhEcznRAHfXiqHmj\nDpkNL3CQvXy6ZMa5Eil7/0myQsM1mI6fvQl9ZEPirrAIJ0oA8jQx1ypBu1Q/\nv0F0m2ceYiD4Bm0UjWPd2CNoNQfbXcVCXXEGl4LoIwYFudtspOL1I/7zul5s\nzjdi09cfoPXNbnDDSyvlgFZXNolEtShhKh8uwTkmtKnWOePiU6VcSoSnxu7L\nfiK15llhEDOTP8DhG1wPb5XqDuLj3JTtcVM1FzpbFKBrvjHhD6az7qL8urFn\nbvijP58b3aYYlPkVW3s6uEStKUntWK/7Re1eZmMCpN4Zrk4JlxvCAEd0vxZK\nADXlkAw9byWRiIN3Pi9ApBk+GuxRxlbOOtN19b/XeLqpAHFkcIXk/lkHrKiI\nJZ3r7QRMOltvsbj+KV0SvTBTQtrfdlwKjSXellhVnA56bGVRffSTX+f5zwt+\nMNfTeZyV0dktRLiYZWTonH+Hhei3i1KcNYrSLSakkEZMI8+yzw4P3sEnhgnx\nTMmDd4ATYwL+l0uQAZfJcSS9kdGKLVBHVgtZ2XNVbMv4iU6algQ2O/R3GAYq\nUUTtFlVEu8YJI9e1Itm1ihGlTUYDd54n6XxlssNG0DVpEJXxUs2Q86AOW7u0\n5GLf\r\n=uqkN\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-dispatcher-4.13.1.tgz","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.12","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.13.1"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.13.1_1619473237801_0.11302255227876912","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210427.c0be653":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-dev.20210427.c0be653","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-dev.20210427.c0be653","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"14cc0a48315820cdd13f6c92094a3ebbadfc6821","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-dev.20210427.c0be653.tgz","fileCount":587,"integrity":"sha512-+XhCLL3GvG2Q4wwLDnvG0SxXExQbvlUGQbyz036uO+6m7Y/RUmaHSoGqixrgR0g6psdaQzVz08xNPC1WfLdCLA==","signatures":[{"sig":"MEQCIFnIj7cUxAJiAw22at1XRM9BUDMETM4lURg9JjQjUW0DAiBzPVPWO0omsy16GfBDOpy8qXJo5Yynsy90m9XCClTdjQ==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgh7twCRA9TVsSAnZWagAAQE4QAI/v39j8zX5TEypvdQWg\naalpnTBDosB7AaoBcTLH7Hsb6ym9l+i7+wVk9U53cfNx5tmkqcrXd+itG8/T\naZ1UFfB2EHk630AehDV0gK6a3jtLL1n6Rm4fgLsfJroRnxkNlig3816BrxPl\nx59GKJYmjOJ74ItQzPW2z4KOX0KbwE+JtkYTDqY96Am8PYGzVV5Z0TrgqLYI\nOPQUS6BUexCbOkwWUs8f1AamXLgel2qiZ9imXDugaiFvlGL19L4BUppaUZIA\n1rSd1tJedHMZRgGLR5TeFR/ZxTSZdY5Hw0k+mo0nu1zbxUHr8R7TsfJT4niE\nSACDuLD8K+1PJnYjpe/dmBtmQR8uSjpB7DzPvDuzophMxxrnMyi2ygo3ibxO\nXjCfurkqxkrUDisTyxJmdizGu/7p4eIC2f8n7X2rPF/68FjymTWx+ctCyARZ\nFiDd5J76rr6vPFda4+iMHBce8ZKbK2CPXhz2lvgQVLpL8TmmyE8uCIl0NGmK\nHT7A4YIXV56uO4a2Jh1dhsqKwSvQJeqg4Q5oG3IeV4ibwRkejmRv9uoVLQz9\nXcSx+cXi7oHS+hD3lds0V0ljlbzLiyyoWCAwt7t68jbEfIgdu8KSJ8i9EJrQ\nJ/O4e3jMaN2Wi4kK+OQbIib9uPyKVWiNiZNqx3tyBbiqRDZ2oR0e0A61spKH\nxUcW\r\n=NB8q\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.14.0-dev.20210427.c0be653.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.12","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-dev.20210427.c0be653"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-dev.20210427.c0be653_1619508079981_0.39996745138493517","host":"s3://npm-registry-packages"}},"4.13.2":{"name":"@microsoft/bf-dispatcher","version":"4.13.2","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.13.2","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"3d392b628fc9ff7c21142e62a387f8e9e044d3fd","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.13.2.tgz","fileCount":587,"integrity":"sha512-Xh+M+EfOaAAJ1fdMklWwocqdiZtA1G8VGPViR01AdwKtpYa9ERl4SCvIT6sWwI/SZX58reiVMYmZ33GjXffG2A==","signatures":[{"sig":"MEQCIFOybp3suy4TWMtIv7KeAOZrqGBvDpAH58EohFbspR4wAiAFWW2qIKzlIpnjxBirMQ21XBO4Ksy2xEFgmD+2A9BBOw==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198631,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgieS4CRA9TVsSAnZWagAAUXwQAKCTis5wtkDAjbA7xYkL\nMe4u8qCHi9XylcVM8KMc2P6zML7T4BWCFNVMCB0vC4rYK7DGHRYah45dP5J9\n8QeKQnDWYN/3GXWGdxVRUsXAORXL0thXl+uNCTpgtZjGP+9WX7wwsVl82Cxa\naxUuHjqXaUbCej+1BGN8FzZ8wqXQna5kSCUJl/FxzE/sAHL2tC14iHqnNbru\naHHGdxmrtuoKj2ae2FRlnX/AzXR5nrWgTTnn6rVoDn6pNqmaAjerjQtIaR/5\nh0D5Bne5jD2fGXLaYVslly6es3K0hQbw4RnxY2Eed9Ne/zcIQrjvU+5FQvGC\nfVbIgPuGxnVbKkyYIPLBVUPkZQbcMo5ImaS/Raxh46mgGBK87k0mVF5vgmV3\nsmyT0ACtKs9vauazbVSibvqno26ihm8WPgVQl4yoryM7q4RLhpUg/S4gImop\nYQ/P+MkzsWVX9MZFguaroHsT9gO8FkuEuoj1/cbZ4Gh7v9LVGabNbyHqMQnu\ngYbWhBUxqmAt0RIs5xct8w5NIUGcwpx+SiH1zc+ozNnILZmZQQSiPxhx4kRU\nar7oOlHeZ5uiHiIqJRNlK171OBNOFzEEFdeO0jTYjT4VSIuKDcyGOhka8qQb\npjuhPuk1kGfmVM2AXExJqE/3h42BYDiCm/eVJ9JrY+wG0C+OVD58yo53MJsx\nAhfi\r\n=T712\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-dispatcher-4.13.2.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.12","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.13.2"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.13.2_1619649720402_0.8809148426502627","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210429.968ef75":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-dev.20210429.968ef75","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-dev.20210429.968ef75","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"bf1ead289640365fd96800b291b5929688eb3ab2","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-dev.20210429.968ef75.tgz","fileCount":587,"integrity":"sha512-stivkliBBHMtUteou3iAGxHvT3xbBGONb+xzabYpKgIUIWIbZETHeedCK9Gj795mmOcaofAcZZ3/NBwVbXmDtw==","signatures":[{"sig":"MEUCIQCEM+rpgEM+WaOoeoJlysN5kc1fTg2muZm49Q24XkcltwIgN6nenTzwqHWjp3MTpV/ILVuRo6z5hI6ZeuNl3csUvXA=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgitbRCRA9TVsSAnZWagAALz8QAKIZDR61gqPuhd5NoDmC\n9zZb4b4BTbQQzWwmPCj+akyLJ//4AOi5uqEJJmqqonVDsGNDhTho7vL8WQLH\nleHfQOt/12V7+UVKaXUbNbNdMIRPmtHIdIxoVezeKf3VlPoIKJ+WiEwH/4/5\npUvU09h9lXDI2lUQaCCtDi+22CHHBnY4tDjMa1/AQK8mdIzETSSXuOxVHKn+\n2JWoBaDmLGWAhwcZS6wRiSWD55dP7IO8wkAXnX9y1fmbL497q4xOs4pyEZuD\nWxxcFWPOg/UMSEk0jsIqLyPPY2RNkum4p7bt+E9nVCH3HBvDSiw32Ed8I1op\nfdp9jpp9Wrs6/Phzqvk1q7vZegGcLF/s/wazJWhE84ZqwZNLekX5jS2SZfSu\n39D4+g+ePBXN/+ZqUj09HPjKWwwwpwxqKzRDX9vbK9khK4x/FW0ekVsjlb1Q\nGribt4+ofpxgpGQMkrWjU/OJn/8I+P8M0Up7G9c7fL6UsMNY/ZE7bve+MgKQ\nHnm7dJ911mjFTnn93eSIEvxAv52htrZYVGYBlWBMJD1Q4Mc3MMWXqNWRpkfH\noqH/WFgNal3UIfClC1HZ+Re8DlOaXPSBSsaDPr2yXztDK35IVF3u5l2Fof6L\nOw5Nr3TrKf6+gyqFVx223wsyO4ZOYSZr/5cxsUIKJDKLe7b08ND9DRnMmPOh\ntvMY\r\n=vQwd\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.14.0-dev.20210429.968ef75.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.12","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-dev.20210429.968ef75"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-dev.20210429.968ef75_1619711697018_0.38924640605740835","host":"s3://npm-registry-packages"}},"4.13.3":{"name":"@microsoft/bf-dispatcher","version":"4.13.3","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.13.3","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"nikithauc","email":"nchettia@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"9397a016295aea710e1dd1d493a6f871841e2394","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.13.3.tgz","fileCount":587,"integrity":"sha512-1LKSU2ECbtiWJmdbIR1OC0+ibKdUFuJKqOTablqMC+yVq4koCreP0cSTKydOTOqytJ6vftW3jGbnmmdvN+9KJA==","signatures":[{"sig":"MEUCIQCGYLsTKtpal2bH/Qrz8FF7IIp/xtR/g7y2SwUlWw/qZAIgK6JsYOk1pDpfQ1mbGYAx0MMbziHzm1PZ976IE5uWo5w=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198631,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgixyzCRA9TVsSAnZWagAAl/0QAKPXOdHOlne74M8sNyGy\nFQ/GvLrTA8LczoMZvK3Uln9pCGohLmVWRTq3uYK7A1nzS1mULOFLyQyK2VAP\nexOMCy8gyxbsGde5wOa5YnT5aCO5/1LbfgBlPEKmHNOsW7OQdgqdw/+LwaZo\nA8CFJ05FnxisYonggHWuw21gmQVzpFxgh+I7+znnlxb2f+GCg00u8G+kiDER\n2yFsVi2GFYBTtinVViK0XTlIBAmP8qcag+u6yDYGNTOuHCdW7R6uW+yKj1kK\nK9Rb3mpPhYYcxE3tm/XhobBVTqf3vuU5ubN8Xq6qyLEEv59wI1CUXT3lKRLv\nYzrIoXDcbzvX9jWICoHrnmSjS33tqIBLudO6mHzaesQ4lpne4BLZFSj5R0PY\nOJFzI3XuCrzbyVw8Khh3IusAsqKKa1q6bX6GYdOPY4qjQRKnfSwKOsRsk7pw\n2oTYXfZLS1NS6m6ehkrMn9volasZ1t24J6+sgK0c1Uy/ysosLHF54ZpQz4y8\ne3VKsaBb+DCDben/eCyNhgUhC0X4YzAXlVfHYs+tUjb8duHQ9rbWM50KTASp\nf46s3N5gruWWAGz9e7v3epkV+bNZWEvTay7ph8mU14cTJCNj5xqCZq4X3WDb\n5F3XimcjHlNXeGy9Xa8TbSiJ3j3jozHp4/JpzUQxAGNWKztB8skH8ssv/ERU\nBb5u\r\n=4/vJ\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-dispatcher-4.13.3.tgz","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.12","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.13.3"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.13.3_1619729587506_0.721029367554028","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210430.968ef75":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-dev.20210430.968ef75","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-dev.20210430.968ef75","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"21606cca285e614e8974fa149d08e1b5e82bcf15","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-dev.20210430.968ef75.tgz","fileCount":587,"integrity":"sha512-65fmBsgTPCSrd8EeaCIml/tluf/Ei1efgeNZ23LNyqtRnccXvjQTTRaHuluGCqi6WeBaPeCrHRvbDxljMr8HGg==","signatures":[{"sig":"MEYCIQC4VDHlWeWgS3PtkIK/NBEpwkJwD9qZTEZ9F3t3nXvF2AIhAL6JDaoyauUWVyyohI8YvOr5X2y1G6hZgQWnl8Y2b+Jm","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgi690CRA9TVsSAnZWagAA4NkP/RxgttWgzvmDJDYZC8+S\nrheZMkI4chIFINreHEzePywz3qlD5YQIAljaSE0nZYQdT3Z1w+/4o6V4IpGK\nvcbosDtpu7DsJFBkZTQB7Pdr7MllvsjvyUhrDdwIdqdpqVrcEwmLWUQpAVNQ\n6j3FDCGNkPARehakX8WKrP4PGccD7hfjXP4nzDXOpw8T3BZhZ9mU8F+jf6Rs\njxOp84G8AzHib582sUHkhaJm4sms9MmAYZ3d0Ne88RTzvEvnkIwI+wNVDRt5\nGq88Pf0DsL3lvkArXNB6a0ViNhKyL5jBu7euvjdNttw4UisUnVGCQUbUCPe+\nK2eaGPX29dqdjb/U1LXJIJlRhL8C8ksm/r1oLs8tDvm1xCrMG1+UimwLS8UP\nWO4F11bKsY7btxHow7dPD98LCMBfPGX+ZyZOndxtREUu0tU1wdeWMOOPgqC4\nYrXwR1KycU17kFvZyLmNil0rY9JdPND8w/3F6dQ0rpQqNwMWvmTag3LeChaT\ny44m6n26MxcFQp64T5F1bkA5qddvhlN8ThLC8WOJPCM1xH4hgKW9AonINUGh\nqSvNtwV1SfDddr4vrupCKXJleI6aolP+dYZL8LpfHDwnufsIP5BHxz/otyHI\nXhTIEE9BWxmrjtLDRWsCKYSjMvPWja9oVq4q5ykgOAn33oA+wqRSHgvLT2r0\ncYpU\r\n=E3jo\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.14.0-dev.20210430.968ef75.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.12","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-dev.20210430.968ef75"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-dev.20210430.968ef75_1619767155535_0.28394394077323426","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210511.07bbd05":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-dev.20210511.07bbd05","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-dev.20210511.07bbd05","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"96841217bab04eee0d6a1e4cf97393450c044841","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-dev.20210511.07bbd05.tgz","fileCount":587,"integrity":"sha512-HJLFOavyoVPBxdVFEG9s/2AeGF43jVhZb14pRZ2OJwGyJf2qFbre4IDx9rfK8bz9v+I5Z9vaWfQOtxyi3UNg8A==","signatures":[{"sig":"MEQCICtmpHbRBUCvA5BJxMdLmXJMCCf6J//vSpIskkbZphpDAiAkGjKjGQF1+V8MkK6GPlevWICW3D3RpPecl5qn2rFo9w==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgmjCvCRA9TVsSAnZWagAA5IMP/RKEuBxh+PF0gjEkjGd6\n7Wo9CF2A2UxIx6QVeFtZNohpDwT7cm1P3rXNrEOVd+K96nRx+XDMxsz4SIHK\njWBhcZ4WU+/khm0uO7I4NbhWjCf2M7NRre1Ow6YJEiL+pOz/FvemEpYhdNsy\nzL0zGF11W9G0ePn1bwFNgiDcfsm6rluuaEnR/v8Y/ZUcgcyK7M1in2ElcuVl\n616CI/xl7xlP9S8ob/hlI55fSzYVbl+R4uBpP6WJXnoDsdLPRv7O7yCIuT2v\nOAI9ZJbTz1omuqUJXxGQavl97czkNUpWsEVEayNGeJ2Z29r5LT7VAR8NcikD\niY2LnpbhhC73ZBw6H7GK+OIlJwtHopVPAzaCH8W1x3dzZIrtM7/GKJJ31d7K\na3cXkD1/Up8AvHi9st06KK13J7tSeKuo8ewHz6RdKHyaRdyY4dySPUShhprJ\n9aYxvEVxIoVwGCExLDNI00yNHUMbFPcXpMvUb69RJn0nX0XXMc/jgeFoK+A4\n86uoU73oGBc3uqDU4QTgCXz8IXn5Ulz+CsLYUZRgtPfu05PWJesoPmoYJNIF\n4CrpKqObVCiO5ZSkHQYPc1SHBPh2OvXhKw4fkXfsleijpbtXyqJ9kMdPJDIU\nQNwtostN7eZJv5crtHBvkRevNj5jluNPhlVg9wfEmp+tcO/sjkRmS63W2Ihw\nRZxj\r\n=NvZa\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.14.0-dev.20210511.07bbd05.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.12","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-dev.20210511.07bbd05"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-dev.20210511.07bbd05_1620717742764_0.27648701905304707","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210517.4cdd4d9":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-dev.20210517.4cdd4d9","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-dev.20210517.4cdd4d9","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"bfb919fd85cb474d94ceeb6ef9304d4523c4d498","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-dev.20210517.4cdd4d9.tgz","fileCount":587,"integrity":"sha512-dyU8FFTFhS9u5Wp1CNtdEZ1hTGK9pV+NaMwVE8c1Cx4+j+YIsTp5jibQSL67a7nlOz/Qzdy7mMj86PY/F3C9eA==","signatures":[{"sig":"MEQCIAE7qb2hvlI4sdgd5kLjL9EBpi6J6fPW0TXo+q0Tqf9pAiAcmbfVZzd17Cq0SAdAZLSkJCrpN5mfDom/SzcOQoDAFg==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgohnoCRA9TVsSAnZWagAA4YYP/iaiITeUjMWpHzml4BNF\n8cyO+hpbLP5+CDmc5zGu69yrHf6inB7IwkkhrbqmCJOtabOC26OAPB/fTC/E\nnVcdc1COoIPfLz7UNoAEZ5vWae1nP1AAe29gK2YkTIR+ZtVVLD43gNfauRSG\nBW6He4ix/Rs6msn8I4oucdwnb59meQk5gH7QS8rCZHVTpByCzy78s1VFbFtK\nzd/ebHBZk0aO4hSs4opBhoayG8N+NRBeEw6QTdze2jfEtMDPDb6M8+mMObSi\nIUNK1SpzqXlYqv/IQVSP73mzQItNNU9UxwGr/XQFgcLkexnUZXOyiDJDddsr\nZcI/OGTzrFfUMUb5OkNS7qjwwvE6ETJXKayYTs6dDnlu2JMQMovkysMp/fPQ\nbyKprmPCowzw2ctyk7iL36vrN0VYiskwGXxgWISFwdJolHOlbBqholeWpbDY\n3PidTiACqD5+Z/kpslmz3/2toXqNIqumuB4xCPCvnTHpt3E+mEVOZudAtU3V\njRNuUWU2pAxm6xF40g1EpL9T+tFP6UZM1STYwbNgp0JOzkKG742BAWPifglY\n6sri7udQfc2ITTYEO29WeTd/A/oNxbJtZ1j0s72WxIQoq4ijN+KiGkxqG6e6\nYeZPygKsJ7M0Y2y9PTgNRT+sPt8GTgO7Oo/C3aaEMGkqn/cyTGZpTxZOZGp/\ny2fX\r\n=Re6a\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.14.0-dev.20210517.4cdd4d9.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.12","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-dev.20210517.4cdd4d9"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-dev.20210517.4cdd4d9_1621236200083_0.9159048713881386","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210519.92dfb46":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-dev.20210519.92dfb46","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-dev.20210519.92dfb46","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"ac739f371c62004de68082790d6f83c47c151b7a","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-dev.20210519.92dfb46.tgz","fileCount":587,"integrity":"sha512-f/AvfTNddrSxiICuJoEfH7fMYiCaAd6pTa8apDIA7j9vaHL39w4pHm6jEk77CXvaedbSN/aCU/3PKZ3FLUuodw==","signatures":[{"sig":"MEYCIQCsbxFX4wmfft+p7Xfq1rLNoczUHbVrhWCyYj/Wj+LC3AIhAPGAJFJgtDKjbYZQHrpC/oRPx/JYfZJKK15Gx60zhl09","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgpLxXCRA9TVsSAnZWagAArhwQAI7nzqqa+l3m23it3PKN\n7S5XQwX0lXYB2QJNNcuguytRjXShhO2lr/Q+1m4f2qV+ao2JcXduVw12phru\nEOihNfQfOXG1A52SPQHm2sOlV6Krww6R7PqKv6bg0vfzBHEh11BohZZceAZL\n/Ohr9Q4fH41u85A5UV1D4woXRh4e2+antvaPJ0dZjo5/lJKbEWSgasJF9T3k\nkiJ0NQ+d+G/ekuEmX6Ir6N08pF6X4OQxpJwhYcbvZhl/jlVxRzIR9eAFdMQP\n88JDgfehEpsJHYnGnZzdn3EQEp2Z5jjXjKLyoDpmeP3OxhPXrRa3Uz0aeILF\nJ44yp/OZ01SJ/LpYa0R8L+h0PFncaWNv/dkrMwkFCIqVM8sFiwEnyyJ3Wt8b\nXJlz8AiSCwf+eZJc75gHiQFLJwK338AiTYf+RpMoLsWJPwI0oVCF2VjKM+7o\niWyKJKnaZ+uhp6DJLS7feEL4EWeOmtpiKlfDURago2GG+h9rtzrJZIx+KNhl\nNafdE4EI6ai8zfwjfI285Iz+C4nT3L+qpHS4Jf/63xC59/7JfD+rU9skuQBf\nt6EG2Y6Z3WPKbcS5CKvlkPDgR68EG6yA1qS7Ja0hrBpuX3NmoTq+IVC4vPlu\niduopm5mL2VhGrL0NwUJi21naI0GWcFYdeyaOIDPitFKzjhM1SLM1RbelL6K\nojmg\r\n=JaRk\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.14.0-dev.20210519.92dfb46.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.13","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-dev.20210519.92dfb46"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-dev.20210519.92dfb46_1621408854577_0.49335972987581256","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210521.64e1d40":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-dev.20210521.64e1d40","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-dev.20210521.64e1d40","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"3e487923494ddcbedb17c4b273217a07004cffde","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-dev.20210521.64e1d40.tgz","fileCount":587,"integrity":"sha512-P2K6wTgU6L4Xiyf3sTqCGmAfvCU1OZ/PIi9GDjMu5OSUZxHKiBFwN07fGvRRjElYFNc/x0U8i6+fGRfnhW/tBA==","signatures":[{"sig":"MEQCIHH/sNTk/ae01Gb3KjakscMCLR98SCb7zTfjHaHkksFSAiBEklsTCUyTKpZx5/jjERM1YpNNQE0MZvGHj0nN8IVXrg==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgp1+0CRA9TVsSAnZWagAATaQQAJQaGa6rC+f8e8T3D82P\no7a34uzq7NDGnIizxq+qLoIYkZM/pWBe8QrvMFdamij5sTlvha5EUX/fy6Ms\nJzSFhcrWUBcybmXJ2D5a3zYaVTJZEugOu1vSnGu99+ynmnVbhjOtOsRs0WFS\nqhH2A4aIs54LF4fVkuLCfoonXq/19BhD8yb1cD3v7VsKHzOUrMZi/JkvMIP0\nC2CowD6Fb6EMUpSMS9lVWARCs8ZjhgapSRW4+JpjyuzwO16oEK5BZ0SZYKQx\nBL1HRGwrOVuyNc/RkP1s5muRYkmazsB/FSnWQvASBQ4XK0QprZ9ZTxtUYTJf\nDF+hizWHy8XW46AcGNKGc3XB3KTQ+Gss9A2b+/LOqvxvVfeBqdGICo13HFwe\nItYK0cqnRLp/qzCbQAKtNysSVhGDx1aCpdgQMoKLwySTAQ5FNePfOWAyC3iS\n9zGcdKPziq7ph7GlZKDS8xy85IiP//R6Xf8V6qgwThLmtd8yHJe6nO2Rr0zN\nOoz//amcE6XsFa6sKgjSBRijycn5PDocynEqpT/TcRMSwsk4w+UwufNiHC9w\nftLj1xLvqUBr+A+WESjj+ole84qbV57aDpebeu1Uae9sCXhVLWaY7xyk0k90\nndggTpWbzmZ1fWP3tD+6+lPIc/vIvx0Slgq28P9BKcSQ8uXZBg9o7u3pBvSx\nP87y\r\n=n3Kw\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.14.0-dev.20210521.64e1d40.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.13","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-dev.20210521.64e1d40"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-dev.20210521.64e1d40_1621581747368_0.6709456810998444","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210602.be805a8":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-dev.20210602.be805a8","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-dev.20210602.be805a8","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"578ef8f06eb6710706cf1eac367f56f1a12dc749","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-dev.20210602.be805a8.tgz","fileCount":587,"integrity":"sha512-mYircv3sRTbGJpmtH2Bi1X/TowUcz/Vv/LBzo+ETM/f9iwZtMYZWxKL3pGT8/NV43mpCzx3BnwVN8pUKnYRtPw==","signatures":[{"sig":"MEUCIFC45PMCiWFotVKAnWeBK9LDjRc+KCp7ETtuDeCJOl7tAiEAgz7HmIgIGJjGX7F+qcdvC+0+c7gORj/DJ3CtjNyYwRk=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgtzJqCRA9TVsSAnZWagAAbywP/R/H3Tqv9nn00BHAKLKG\nc7+X1hKkHwUiME0IBCgdJXSGIlzUgnTH9lUIIFDpPG5QqoiyYRUee7cW2CAL\nVo5CRe+z/ZAY8SzRskfrLX75He3W5e6D14MiwKpNFS/YsHjv9mVqyhjUIz1N\nNQ2PQkBIoDccGzXPddWniBu+lU1HzBfavdW7g56SCsY8JLOWuCnTgz1gfJL7\npjVUtGOZEa9t/1xDEV91ass772SFJSAFcGvbq6a/A6XqrYI68YVdSGEEzOci\nmjXqDolVmJm9jTYNeDY9W3neN3bNvJXU0jOsXrS8GBJKW65Li95OYkRbC7o4\njUCQpBg+7mf1f+b8+qjfkIktEONdpuOd50UZFpT7tPYb8GYP2b+skHu0Mg2W\nrmD4MwCEQL4djiozgt1cF3V1euwW7DaGojUgPw/UOF1mNiSWlH2IZyZLXQE3\nbx0zNkAWxyEk4REVM0PVBgESHORmP3385gyhjRy42QMMePVR9Rvfgj/glva6\ntwT1cOP0/6MtAH5PB0IIY1v4Q7npSOVZmwO+L/0HCpSC5aPBIIVIMUnMnlw4\nMC8+CFlAwGgLmpXVSo4BL4fIZB6JMHoqwzyVjQCKOp3NP14cJn5QW9yfELB+\nSPXR+E7KXolytq7JnDOk++eRh8GogRsBwJAr1n0MXXHPVO7mDKSDBUYUlJzD\nN9yi\r\n=o8L3\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.14.0-dev.20210602.be805a8.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.13","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-dev.20210602.be805a8"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-dev.20210602.be805a8_1622618729495_0.7147793594875693","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210604.3f9ee15":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-dev.20210604.3f9ee15","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-dev.20210604.3f9ee15","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"12ba69bdf58f92714374ad342c33f5facb8d7d39","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-dev.20210604.3f9ee15.tgz","fileCount":587,"integrity":"sha512-DRSW9FOAra7xzpYeAaNcfHAxwbDa/djWJQOiMN07oVjJX3JzVRqm4S/h3XYN+MAmO0HwAFezyTYNjPA3sM15sA==","signatures":[{"sig":"MEQCIAfamjL6Lm9qSxKSMh7z0FFR4Hc7woQM05AZGy3fempoAiATVBe4lEKX8XNDNQFNQBMkazjMrQwd78QoQAdxWJFyyA==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgudTpCRA9TVsSAnZWagAAXAIP/Rg9dBIp66yOFBxurrSV\n3CxbfEfqpkYIkhHQIgZEZGumonMEBYXk++tqoHVu6OlUwESp1cXsbeo+qaRh\nv/oogQpxLGuVZupNHqhQc+6T6O3D8H34vTgUutm4jlE1KO1IC/gAczlngXU5\nIWsRHNpercpZupmjCDeV+OUoHwTWpu4dRLi3ym55EPkXuTB5hMGNuDG4FkOm\np422VwWEA2E5CJlOLiOqqJjSiGyI7gS5HWmGty42scogeOpucKpV4w1kQofs\nF5gP9UsYLSVOPmseAw/ffg3XsKwCp9FJ++mu7aGwu6yUNI7W4pjqVaqSQfti\nidwct5xMVX6dRmlYKY5ohayhlux5Zztyguhxp1cIga6xuWnPkRXYlze+FU9Q\nuXtdBeEsvLWNXbkVm+fqFC7M0vktJvilSU3Z+/rnXxE1XvDDgfU8Pz2RKQEX\ndfla4O63IOkt5QlsugGtX6A6OGon0kmLKsnNJNSI4+dZx2VlgVuVnprv1aVk\nKw0f8JlSxF6i2uOt0tAd779oMFuKhudhCAZd68bmAdKCgEaqxGNYsKHUVD4l\nXstM+YeNB/Nx13bb51FX3vuvOJ5DHgYtSZvqW8mmmDLx98sfBkrpT77ZDB+V\nLt99oPJ8z7CnxiwGBzAP8kzjS6Tl7lqSkAsA/TqOVHkq6H/2AJ8KO62hYYdv\n9QDN\r\n=P9UC\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.14.0-dev.20210604.3f9ee15.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.13","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-dev.20210604.3f9ee15"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-dev.20210604.3f9ee15_1622791401660_0.7614048885356179","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210623.37f59ec":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-dev.20210623.37f59ec","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-dev.20210623.37f59ec","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"6803f0cc53cb8dbb9dd09f9101166eb033af31b3","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-dev.20210623.37f59ec.tgz","fileCount":587,"integrity":"sha512-/EmAk187eXOdeGrz38rQfbnuu2rsKOwxBEacRw2uuo2PpxenbN4b80/htK05cumKbX6unBXLXQLFvtfvcjYCCg==","signatures":[{"sig":"MEQCIAOUKJwtlHjLMHggWQ9BTAJ6u4gJMHRLnw4taDm535PuAiAw0EkH6g46alw0kbgTxRJFeaqZtZ1xfHm+WeWYS40ZGw==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg0uMrCRA9TVsSAnZWagAAAJQQAILJBn0o9lGBsnERjE0R\nMaCMD1ybSjfo9fckYxeMJUU0D9VIZRTqgWeBETIWCd1pFsrztLCAtMYfLbvp\nue8D9yox/AnbAzFExYAmMiGt5hO9IwxkGMOxNHSJsBKg5ZUE7cCq5TQ4/YdC\n2qCF7LKQSnHw4Lq1ShMoRx7x71bIOgekRw9mZH4YQGOou7+cQN1agDAXDF84\neN0fLboTT8Tw/Hcu7Ztu84kuL3nrrqSYIZd10DUMT+siF73eJtr4KyCdVD+x\npvnUGLWJQSyc4fUDmUCsBuRbXtqJJ9AeOFKXJMzp7p5ouEbd+4x+GzLZAdDJ\nwIVL+p2I+ar/21z147riUXcWY/D2zZJQbUry4T1JvsLjzKXO7dOPA/SPK/+n\nhTF2lDMXZFgITKrYQh0bdELnKlGfi+/uTExCaE8nH9I5vJN7aZp0QcT9cDI4\nnEWd6MV+vrmfNmsJl4TIpnmavweNHVAICW9sfU/P3unmn9QCaLOS2RgCDc2W\nHh7rCLJdxI7eXNdAosrwDhWRdC2CgKSoN4jTjmfupT6d/odKkivVM/4+PzrZ\n2zK9brVK7hSrC0f+R4BkwR9pg0i/OAMT60FO8jG4PF04XlapnkWCvRsejGL8\nP5l9rMeDj0UdEOn5O/Wmxbtr+yOuc+VH59wSiw/XkFV4Z9b9BGVUTpV+2Zxf\nNsiA\r\n=U10q\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.14.0-dev.20210623.37f59ec.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.13","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-dev.20210623.37f59ec"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-dev.20210623.37f59ec_1624433451268_0.15892494332764517","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210624.eac7a60":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-dev.20210624.eac7a60","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-dev.20210624.eac7a60","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"66538c5dbbbfcbe51b336154e7c1b148f4ab7078","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-dev.20210624.eac7a60.tgz","fileCount":587,"integrity":"sha512-SH3W5HVA3C1kTWbqxS4BJQQysVChIc8hS9AbObhOMqMw+d3v8TKll7edsdONsxqtghB9pL8SVF1tsXqPbSUmyw==","signatures":[{"sig":"MEUCIQCdP1avR1OaNCt8+RtznMkuCWIuMeScsGSiyAk+hxnhEwIgIjGPFoLl5LdNIGQOpSCOJapz3B0OdVgEzxKPIrRzO34=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg1DKmCRA9TVsSAnZWagAAPp8P/0jSwBLfD112cQt1e6AE\nWu0uhfAOtEhqwSkQ5lPHhkdj36p7FBDW/AUBzw3LunMN5ixl6hNhm9Y5V8M1\n/eN+xB610Va4tMWUSRyGvnlKwKIlVRxh71pSRxa4pWPo8k+EmLEpP+laJHzg\nIazd+7050z9vt/daIf7cEEPPwfWAeYfwsnvrn37qyA0002hSlYRShSwymDWO\ncQZs6ynLzwOOaLorglQ/t1WXNCGOmJjZkqKtpyPLpEh0f6iLuu2kMAgpgSyQ\nI6uGaQgTZDdq0//aewxD/Ksshu1DAZGrswq5AB1iTmxCHWWfGLeN2zOVH6Tb\nh4rd31/NlUVD2Fipz0yUpsrspvZAh2KC9+FNQnFtGJMr7FtBIwvO+5nDCy7X\nefFHrxSyc36PrIk4Vry6Rz5Uc/vXVI8COV8/WsHYXk1jarQ/ogVhgkyGbDq9\nex/BN1PqvnzMUGv3FekAxw08hEy57tFfT/QHYm9mk6O7QxUSASdbLhJVFrLC\nymbbA8v9nYu7GiN0c7KMovUZpjaz1NCkVm7B1hnbQTVFAABj4Bmsl7Rt4r4/\nl/gkiGXHFtrgJkaZZEYWIXlzHkKfphTcf7QFzBroBp6y37RpmMnG/GyqsMOl\nZdopovFNwrQj3Ir6ZNZQy4010rq554WfCdAFkjbek57I/T/3le2Z1E1JfYbm\nkeLI\r\n=KmFB\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.14.0-dev.20210624.eac7a60.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.13","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-dev.20210624.eac7a60"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-dev.20210624.eac7a60_1624519334094_0.7261857033169108","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210625.132e0cd":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-dev.20210625.132e0cd","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-dev.20210625.132e0cd","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"8085e6e290d69543c5beb6fc53c6b52eac99b398","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-dev.20210625.132e0cd.tgz","fileCount":587,"integrity":"sha512-gbOMnYIzqu8qIqjm2CCY8JQtnhkufym0OugLzpCNFDzTCtZVZdNCYKVZiu17gjH/z4hM1JNe7NUNb65q9Tlcrw==","signatures":[{"sig":"MEUCIQD4kFhC3IFmctULdgpikaUWViU9NF5Tn7ZWEOX1eFcEwwIgbkt3m2nCcb2PqCYjddO6pg0zn5onSovPdtN3GyNJFec=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg1YPqCRA9TVsSAnZWagAAxyIQAJBGwlWFs+O/l5QyWELB\n6SW4ab6OmPmLOgGLEfFEzCYCAeZbKXF4XQh9vcxH0aoAkeEJyZqNw4xj5oJE\nv72kuVmVxDbri6lEoVFAFtuQBpTm3cgsImCCOAmWf1VkyrdAlpyEBK1S8KIN\nn8ZRw7gx2uUpzuZHam6JpcVF6Zam6ZfdL6vMu+I0ucemy6HiV7/uvYmQ8OtG\nZXWf3xukKTDCtm2y3qw8m4zkFkDNgPRJbLYgHQyQVZuaLBW9z5TFzuuQ4eHy\n7U/I0ZoE+bjiP47TYCywQ5lyOwXaTAMIMw5YcP9Pr+etTZ/YjjRLwZEwXTlp\ncWshxhakXlYUl4N0EnVhldtFj7fd4QfU/MoOT6VwKsTc4Ve9vx0Y1vYgn5Sl\nseJnBodQ+RL/49uqRMDLh7y1s4iLRIA3U/C5j9I6Scz3B2VDKe6/QzrN8YKO\n+iXm8Tlcw1qbQwS5MAFfWE2HfcgSzt+oc1M4t45yAPbaGCXfiTAMEYqreZ2B\n3oeqm+Dq+JlxxF+e1bK/glPCeKXt+j4UBYvYP3Hr+1MgpbjmgIJ3eQD8KH5i\nfZZat2MtVrs7cD0GbB5qOgkadtjKR7Zi6/WyzDKrVpOLfKMfXtqjCpWeaGGJ\ngNVE1bBejUfvHYQzakc4+E55TVRfocDTNPWmsK4tL0z2MTjTXvXhHH1Sbytu\nInqU\r\n=KsMZ\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.14.0-dev.20210625.132e0cd.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.13","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-dev.20210625.132e0cd"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-dev.20210625.132e0cd_1624605673492_0.6386564964929764","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210629.e97e6b0":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-dev.20210629.e97e6b0","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-dev.20210629.e97e6b0","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"b8cf03d8e76e3b8afc5dd7cab728f2267ca0506f","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-dev.20210629.e97e6b0.tgz","fileCount":587,"integrity":"sha512-8OuHws/cG+Z6HXSCiWF/VKur1wTSsUghkUEI5XqwtADVQ2+RW3+71WIsg46ujh0TO/YR8VTgwgIMUqp4IJWzLw==","signatures":[{"sig":"MEUCIATJslzVtXz9c7zrWatWpiN1UO3/2QgllA4yNvk4Nd86AiEAixmrqpYf2lPATJWqglauiQy1GIzFSH5mqBsodHkmJ78=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg2soMCRA9TVsSAnZWagAA7SQP/03uCCDLu5b7nFvVmpyI\nV9I9TraPwlQT3r03uKqsTfgO21/iug7G4H4Yy2KiGY9DD3k+ZMPC3ye72RcU\n3ORIej3UQwdzh9M6Uusl6tf0/Jj5NqkBsagM/UeFn28M2vnUajMhxGqos6W4\nk6wlXW7lGmnrawUhUfiEv1ag+2xwjRoWaXKxhu64DvSsNxY6Bxs4oH8fANfc\nSl3hDkgL/1YpIieX6GMFcoTh+GsImtHF57LP41FNmu18Y9rKriFlOpRbLLOh\npdudMjQFvKF5AL/rN4alfQ90RrSA9IoAWvIH1t16e8vCjMWpFjISXwPLvhWX\njRA3cRMAT9h2X3ETx0qNazVxe7nocpR1pFr9NPGnG1Tpc72R0yNH95WTF2lT\nJh/5zCIT/tfRM9s531qjA5kiVvfaKME5byNW7WqWO5YUVbf7wc6nq+qyn92b\nvfohEXh0xQoChyyRwspDwUjiiy4vqeTCXcrfJ4lWvcOEX7OD3+5HOBOhFRbl\nJCLs7ER9cCoRG5e41kfcNaVHWFcOIUWk5xr+IufDkCKb9lGz644HayWlRRS5\naftsO13siomyIpdGEn3Kg6KmjKEJCmjcF6zWvSkcSR2MC4+cT8hZH9/o3exo\np4+mAKWO/F43cnlVwP6ozrqp4+zn+fjdX2mlXENXyDRBYMU8RkjSYpVmwHkD\nBDFd\r\n=bXY1\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.14.0-dev.20210629.e97e6b0.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.13","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-dev.20210629.e97e6b0"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-dev.20210629.e97e6b0_1624951308256_0.35506152028289817","host":"s3://npm-registry-packages"}},"4.14.0-rc0":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-rc0","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-rc0","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"79d50cdf7d4e01b4a9bc2135b6d464573d937720","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-rc0.tgz","fileCount":587,"integrity":"sha512-4zJbfZjKGTpW1iBSCCvr4buHRI9j8j9RMIuAKSvt5pWPkW3Xcgg/nETIQdA8dJOaVyklRV/GsnCzNVBLenYywQ==","signatures":[{"sig":"MEUCIHW0peGgsiQUNyvn5MmNLevOjJKcwZUJQj2azDBj+n6yAiEArStlzm8Mk6G3tCtuRZsXMVPTgH1MLjBSrHaDgA6YP64=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198639,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg22EICRA9TVsSAnZWagAAPz4QAJ5EsQczZh4U4PNyJH0+\n3cTZ+ZLsqMi1yrUg8IguwKFV9+LWtHcLB8J+j7xc/ZQf29IDMjOLg41YJYaj\njGSUUOlmQipd67K8U8XxGVksPqvx1BXHQ8ZGgwq6HTdBthpEdmPk2Q8gzclt\n15L2KO7QYh0mgwMYHJre/MUjrArzQypM8VZonK+XtVQPc8nHSYZTA2rbP7i9\nZWXjyXeI7uEay10lnFogDImLUCRqZQOKyKg+baQKG/ZcI/RDk62iwBR/nhCs\nHGOwb+ErlfduyMecaH02Igwb7rX1OmGG6mlSzf2p+v0HqgH0TIuzdK1qB9u3\naMeACS55/Fk4DvakLtLqdGAwSGfENT2V1XRKweeQwxZeuMPMYvq+vHwYxUtq\nz9n2Am1LOFQmy9yqyOkw1dnYPDf7gQwOP9u9EvJi6IWllpvp5dDEfg7Jjrk+\nbkcd/qEXRVN1ONwupCc3VbEqEM9944dnbxQn7ZV2VwbBJg0IQAOB9UyR8kBv\n9YGCRvHhepkgssT14a3iG8TcJ/FxbLkafacpzACiL2AJL4fCSCYmsqrNOcAO\nBDTaB+YkKRhFfqwzzd2jhyKwHEQGYVHxNW/U5qFyB0GGdeJtx3gPt0U2B8Cy\n/WAviaJLzeP9HFUUbfomc9yGMuS39L1W6orIpI+MIlGr6Vw/BuUttj1cZtkX\nWoqz\r\n=Xe+1\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-dispatcher-4.14.0-rc0.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.13","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-rc0"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-rc0_1624989960082_0.9917776022979741","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210630.8078cc6":{"name":"@microsoft/bf-dispatcher","version":"4.14.0-dev.20210630.8078cc6","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0-dev.20210630.8078cc6","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"38906a3a97ab87f6eb5e9e0da6edeb2dedbeb990","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0-dev.20210630.8078cc6.tgz","fileCount":587,"integrity":"sha512-D9YOh1xOgXGsKCpd7WTaRnwxCi48LRkZHW5oDqTsFCLVk7PNZ/YWMx5gTrkI+iAmH+gKgE9BChliBE0w0BFl3w==","signatures":[{"sig":"MEQCIBUfPVjTMxLSyL00oX0DFw5ONeOAqeZIPBUW1Rm7IYpKAiAWKIiRJ0e3azNfBDvXyK6JrtEBUFQjqXu1yOiQM7SEVg==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg3BtyCRA9TVsSAnZWagAAWhwP/jDxoKhGBFKkpW+3odzQ\nVd5ZCxfLj4wb9F/l9EkL3fu+IZ0UBEA5rn2K4kBfrQ8v36tGQGFUzgyhdNt+\nphR4Zi78bwl45emzKVJthZyS3qRdghG8sh369hrEvzCpP2alG9RB+w/6YZ+l\n6pazmMrNOdNoLfCw4S/LIdHYdG6X4IS5ZHf23dPLNOT7j5gmPI8ygW3m/L02\ny6360rK2VnhZgsTDrpVMixpyW8KhedR3vrozuWYQfYoMbUB9GBl0aaSrmLiW\nS/bLCpiksxjZfmYm+4XkhDvRGWaQCRMB9u7sw1tZZs/TkHJ4ZJFdxFKHawyG\nZ+PdKz6hLlMetSXeowAtD95bW+TdAvoDooHupIax+tMM/UO52b1Usklc2CNz\nX3hia52tIOeMWWrE+QP4DmvMWvvsvTYjhuQIQOBIiuypNOgB4vvDIttvqau0\nf5wlfp/R88mrRGystKz5SaDDAPvsqvZEAras5GaPUpMb7uWSAtfUMUTDqu62\nstBXbTH1o+j2El63g+OjeZzOKkEWhSkt2w+qGLg7cXAWiNxEHbSKgDgB35yB\nYNOERT91+EBBtr5wIRUw2luHrSkSdKg87vVz6lbU68RTcOd2E1LP5EMYIV8m\nxr2WR0uZp7er7R0kufshzIZzmLD823lD6cYVIRjaFTOzehdRxQGICtoq12u8\n+gTk\r\n=5icK\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.14.0-dev.20210630.8078cc6.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.13","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0-dev.20210630.8078cc6"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0-dev.20210630.8078cc6_1625037681666_0.3358858278351311","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210701.be9cd6c":{"name":"@microsoft/bf-dispatcher","version":"4.15.0-dev.20210701.be9cd6c","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.15.0-dev.20210701.be9cd6c","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"c62c04723e7869694df1130bc7aac20f56aa1d47","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.15.0-dev.20210701.be9cd6c.tgz","fileCount":587,"integrity":"sha512-pZip7wf5FRyWXtFWJ6j07bZk4Hw8pIfiR+AUM5k54OnsXFOJEc/rW1UhvygvxeLAKfo2rW0Tj6EJspE8RG9DfA==","signatures":[{"sig":"MEYCIQDMPfCwH+L12dwvAAFrZ62L4L33viMyEMmm3A32XfRHxwIhAODC51ntM2ypxhiIMGFrGenAw1MeqtZ6IRuZ9hjfs7NW","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg3WyDCRA9TVsSAnZWagAAPHEP/1C24xvnRWKYGc0cFvXk\nMqceRJN9Aw4W2zsLYVdDi6J1crRHiZl3KSDTsorf9l9kLK8LpJQL/uodCA2V\n2Ipb3wUFZ/i+TZvNxoxm1jk0WtwRVK9xuJ5qQGZZB6FrcQSC0/8ddpytdUn0\nYjxxusKWCqyvgeQC0Vq6nwKkTKJxga9X019RfFYHSvA7GFSg636LXzNeChju\nPeIfOeF2h0BEIOYExq3Sn/Li+OkEhFunB7s4qBJwjn1xkS3jLGV35lqeZk+c\ntHBvq8rGwZI8Rz0d5v0mCmN/9tO3ZlGnoV7a4CdFFK5GDP/AJpDG8Nf8264A\n5slLHb2D+yp2FVmX9+Xci5kh7JJjgvS3JCHZTqh9pR+OFio83RhumrvR/mO/\nSwTFlrlMrspY8DQ464bflrhecx0qlKMrueW7c1fOGBT0lLnT2KTUTf38WE4P\n8EM+BWzjaG7LonAMVa5dWejwa85iWcKR4wvt5nZIdQulx14G/OdLRZpckRgs\nxckyOQVbZL9X8fRKB6FBGt/5rjzhCcSOhL3iWstC3hEj6+1v6mLBDAH3BeSB\nkALQMr72NTf8+uHhKrai2P2xKpKQfY9SWtpjdooMgexBstRTyXLY8ebXldcA\ngUtnUxki1ctj7Xo+5nFStCM6Y85tNSHBiwG9GvdqlVtPtDtmVIxydF48WimU\nE8Ka\r\n=lMTa\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.15.0-dev.20210701.be9cd6c.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.13","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.15.0-dev.20210701.be9cd6c"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.15.0-dev.20210701.be9cd6c_1625123970633_0.6150451766398581","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210702.cbf708d":{"name":"@microsoft/bf-dispatcher","version":"4.15.0-dev.20210702.cbf708d","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.15.0-dev.20210702.cbf708d","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"418aaa95861ddfc0f8f0889d977640f11736ce9a","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.15.0-dev.20210702.cbf708d.tgz","fileCount":587,"integrity":"sha512-TkzWsWoBERyIiRqkKlZtCgf7VZYwts5yvida+zufwSg4Z/Y37Jk/LYKzFxMzgeuw3qvBwrODv5Gf9HV1rlIh8A==","signatures":[{"sig":"MEQCIF6+tkLyEAseIrJ7ZRjdGW2uh+m1/Wt12vZcM5UBlks5AiB9k5lEIaUJ0hEsF1W8wp2MLfdhhW4C/FQj9Fpx6WNMXA==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg3r5pCRA9TVsSAnZWagAAK+4QAJVLfEqomkz9RjW6r63B\nuK/U/5pqHz0i63QuP0sk5eW4TLZC/P+zflWvhCgiwXwYAqDRQsOVdvodVUju\new6QV0em5c7+6PKRuryKH+NOfDdZj0RzUYxG0aT9960R+Jpktv4BQVFvECLR\n64S7G2kGrEf0mUn6KqucQhWMrKeE97+lM21X6FekXDSNKLiB9bIBqT0vllsJ\nBCN8ovDUx+E2Xs/WR/9xT0P2jUEy2L/b8kKhtlKVM/3GqxxXoa5PyCeGT6cR\nfeeXIoDyvWqRB7mX7htyTaOADKlFxYVpi8RCBMpCfFe9UDx1md/zgjpUEJtQ\nX9b6EL+bC7iiWhlYi7bQll4qkxJ5Qnuwt3CgGGwV5l2hT2DFUjahJKsQLaiV\nqERF/yeeFTgmsqBjDgxU5n93ncsayR67bc1Ukn1oUdXijoMr713O7YaOcKeD\n7Rrhv/C/xMDLc3BjFG2hGwzaj6bkwAYDPfgoc9maWvH6MLTGgBLrZlRjOOCt\nudOnbn/q4U3d5IHOjiDllKAm3T0ILDwgH2gxA7xAdnrxlCSHoAk+gH9pZL5w\nJIPY1ewmj6erGOZmrzmmcdMRoe39m8DsP8plG2MYjkHfRJB1eTe2ozjqChiO\nfpSKPiTqpYv3go8KIO2sfJr1VwfEBi+7eRH5FF94TBqDBE8sfkHmaIda8tLx\nioGs\r\n=TLdD\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.15.0-dev.20210702.cbf708d.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.13","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.15.0-dev.20210702.cbf708d"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.15.0-dev.20210702.cbf708d_1625210472466_0.23075738117283673","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210707.cf471cf":{"name":"@microsoft/bf-dispatcher","version":"4.15.0-dev.20210707.cf471cf","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.15.0-dev.20210707.cf471cf","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"4cf9ba3b280f59f2d2c0359b291f6854774d8542","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.15.0-dev.20210707.cf471cf.tgz","fileCount":587,"integrity":"sha512-Jr6G0Jhd/PvOu0xbtTLxPzHijSxSwKifjBRjcDK89yWWs3zwZA1RD7jddg/VII3rVspjGONKRJyz1A1/BTHA+g==","signatures":[{"sig":"MEUCIQCexmtwouwINoMAOJm/psfqJlCUSJWr0xiflGkGb5zkUgIgRBJpawaxZ+aahHnCpegXmzCK/lhd5gHJ6uSCe7plsZY=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg5VWmCRA9TVsSAnZWagAASNAP/2uqNnjXca4xH43/nuV4\nMYu51NmIoQ6jlHmnACLo/ktLULeMMO5oSJI1mbxAKFfMNcXcErR2TTnXeq+C\nxoiiXssy12mKFZ5j/056o4gxdkteTtDUQyWC5Q/l1YxFqPcxsifP14hQrvWE\nWHAEY8RjCHbC4ZeQvKoWn/gZyiVK6X8V50RtLHLPCCdLGgqfAdzJqFqqV/Hy\nOyOruOUB7y6Od8zNsuAyg6FdCRcPlcFQ4bfp//Eyd1TeILrcPvol8NnaerOS\nm7rT0amecdP3exFguZaLAbWgdzxNsFYw1XIaD3Hwmtx8WzmQzEYEBF0Pv0ro\noaitbgvAR0YBUwMpoZc/NPCUkw5tEI/9Slsir6XUCgufE4D/rmEqXr2Dvy1a\nvCUADhnFlWhGnfpb9rEeq5H6kSaRhmBmJx/9L3fGHd8FAQ7Pv0+48YiIoL/S\nNb+OVp9x/1+MOicBCYvCgdIFdO6EKw2riBzxLwdnzIDqym7T85OouhHSnn5w\nTwedLtsSYDl41Vpp7eXpJrLhyf4pej8kmXTL30VVC3jrMSyGR0/U85Cfsjjs\nqu0Kt/3yL+5yVk0kglfkFhUlV0Vm+UHDx36lzyqzldpfNsKpB5hhj51xMLwt\n1YjP38x7sq9U+rKA8j9LE7Y/A7RTQea4ERuRJUWa7ouLi4lETknD/4CPZUgX\ngQZO\r\n=Mu02\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.15.0-dev.20210707.cf471cf.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.13","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.15.0-dev.20210707.cf471cf"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.15.0-dev.20210707.cf471cf_1625642405968_0.4264964747218647","host":"s3://npm-registry-packages"}},"4.14.0":{"name":"@microsoft/bf-dispatcher","version":"4.14.0","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.0","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"f53ce436e1167f3c1246015886b404a1cfac6cec","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.0.tgz","fileCount":587,"integrity":"sha512-tPDHtuXjyoO16yiVcLFIXSMwKjZ0ZVs0J2JhhRnC3Fee3WUIt143DoVpPa+djSBk/eP1SO6w2UFYZgGtJg8ySw==","signatures":[{"sig":"MEYCIQCElrY0jNnXL98rZn66US+KrKfpSCCOqGfRg/pJ73uNEQIhALTB6ZT516wct4cWBr/+rJ0uUocqr7VmKvtTvLfetSBk","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198631,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg5gAACRA9TVsSAnZWagAAG7kQAJxHSzDK5D+FrHpzqqS/\n+i2ANE93ofgesj8BH1MqUTB4hN+3wdl/iurdZIyDOs3WehMzqj/IOb2KKcH7\nglWWcOEbjsPn1J6cb+RwCj/thDEF06lJFiC2wNlY8tqabWprLQwWGFAn0aqf\nO5pXxibo5X5nNvxoXg7ns1kuoiKMaAjepsWu5yOtXCIlh3X9v4r7B6dXR/pk\n3rrkbTLbdPwRgy46vBYcZD4N7BpG2e4kcngdFtRkA4i+HlrjAZbWYekGJxir\np9kOAtd2seWj7iU1z4LZ2LqeHPub7VZVCz7c7urrq/Vt9tJA66mr7/kei02a\n16HOQXzreN2XP51pz2s72ztTRnfXcsb82LsPTsS/Pn95gwXihFRdpQKTnyoG\nRp5yK6SY1PZC8C+0VC7TOAs3VjBoK51ebVvLOwe8/kc0aea2VTPaZfY3zMxl\nh17MBE1GRP7QXM6eX37eXch2j008LvEKHFci5FbDQPvcD6Y/PSXzmayTBYqY\n6nekQk1PJwvHPFSWnTsKM4LTjrLyBnChVQJMk+ucukHZAgh622yagtVmEIB7\nqoDR/3SEDHZz6TK+jKx7To9eS4KOJ8HINdwS1Fejb0hJdGb6eOrjBuwDuXRS\nLYzVw8oYiYYAyWK9UWtUvlNDmJlIsUCk9jEwaRD1p9f/eIBvKoFQnfyj3wI5\n6/6j\r\n=KNlA\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-dispatcher-4.14.0.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.13","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.0"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.0_1625686016294_0.87359661233256","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210708.f16ca8f":{"name":"@microsoft/bf-dispatcher","version":"4.15.0-dev.20210708.f16ca8f","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.15.0-dev.20210708.f16ca8f","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"3b2f318f3e51f48ddbf1e7dfe1a3abd70f970e1e","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.15.0-dev.20210708.f16ca8f.tgz","fileCount":587,"integrity":"sha512-Hu5elMDWGCSB4Y4leamDqE22qFAQybw2v3ZgGqSo5QPybuRrDzGl8VSbltAdkfSLNZiBlZJlsBPuY8zY1LhmUQ==","signatures":[{"sig":"MEYCIQD1lyctcZaEQiW2w11feB+YimiRcHPr3c0tnmO7Sn5zGQIhANKRGQImtD9Ntp6Ssfwb0FEDy5B/GGZ12Hvz0mnXBojP","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg5qbwCRA9TVsSAnZWagAAQnIQAItOy9lLNwskz2TadA6w\nka5M19nd7MZPOrpGEhj8rwlJkkeQnXWUVYVdLhkPtSgD/pa9nSnJvCBIjGDE\nDXwtGA+Wk7pKVj4Vli6IRRQP5xq59TW65sQcMG/kzao+BSmdoHjAyLsGKYOA\nIzFqzlhDDevRKC7mRKt/dSQuQChzUuZROGH9rQMQTQXKABWl7oDQQr3U7hNZ\nbchjbUfs25P94jCE7ujhwrbKCCp9LCBDNTotKm0AmsRa07xBymnwOyruGz1w\nZhIWQYdjH1z75vfhsBtU3bRqZmG5qhI6+AJZSpEkn9a0Gcbthzt6BG/Wljp3\nnRcCt/lv/ny97lJ6k4nJk6t/nCbduu5FAtt841TEjgpGfS4yoAiOenJRjyOJ\n4/byucsRRs13mMUwd+Flf32eaP3my0OKXss8QABS8627LD4cuOztcW0NJSKL\naClnp47HSOG9Ob1PwpKV8vxmeolCyRdQL8oRCWk1gBZ51xV8H7fc9xrwL0/Z\neP4Sv61p31aCRx2gRPjSHf+MV8LrdjPw+If07HnWMpE4WwEc67kuyTLB+OvR\nI1e+7xVmM1kJYcbnIFUeqGFucJ08BmcgLe9e6x0VcS7shK+KkLxA3fZGRcMg\nCVJSzoO4ytLcY0frj8stomMAw08C3imzAe6gS9+BR9FehE0D08tJ3SwtG0FA\nLgfV\r\n=cNfU\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.15.0-dev.20210708.f16ca8f.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.13","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.15.0-dev.20210708.f16ca8f"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.15.0-dev.20210708.f16ca8f_1625728752551_0.541264167838162","host":"s3://npm-registry-packages"}},"4.14.1":{"name":"@microsoft/bf-dispatcher","version":"4.14.1","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.14.1","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"ed5b51d0b942f63bf27ba7f5c88a2a37ef78a496","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.14.1.tgz","fileCount":587,"integrity":"sha512-x9kywbgPx+8q9dlueBmnVkh9tQwSdn6jf9gNde+IAqvSCWEvkt/IGcwjTDc91YzMRzUjnSEgrtV6+5BBOoqnxA==","signatures":[{"sig":"MEYCIQCUyI3QKJNbptCpWO4av7kVjLhSuQ2cRf7MqthU5Xp6igIhAOKaEsn6m16Xji6YEvzKI4insK2aGPDdLbPk0OBVsyls","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198631,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg50agCRA9TVsSAnZWagAAlyIP/R9WIz2Wv9uyT3wvg2vo\ny7H+bfk0PPokfyDRqEou8LjE7RWacdM/vKLHuDGuhCNjgnFoergiQyyci2RI\ntEl0E1rXEvFolZe8JaOGnDsAC15F2NDrkuHl1Fo//VE6x45Hj5j+5tgkhtbN\nI1CCkeKu+Y8iUQPhUXCI3ZTwD+V5OLHgE7dys4gNezPzY/PR6ZRKOsJnjFyX\nZaezDiw7Pv3zHfFC77w4cuMyGNx7cy3sLGp++2fuokhJYN7E1y2/hkBVT1SW\nvzZytdMoO/tbSHgw5LVFbolatN8LKgPFZ2FgWI2EvhAkX2jKkbfr2iP/18pa\nbuP20olOi3/v5C3iMiiakesTz4bv7pGN1k2t+rOLmN81pI4LC4sysIG5EKiQ\nImDPqxT6XNkPwwewG7aXAjHYEV7/XjxQX2LiqPFjZ8AWtUchRTxACYynL0KA\n9Hi/Gs16D877kocttDmjG0iJVhLOq7fWxpUFrbgRC5U5WjdikdaWPMMPI+Lq\nswx7fM+O4uwX56tdU2HR9bZYQSMeHOEcYKpnfShP9hFqkfT0V7dJMFbxWJy6\nUs+K3sJGku+7+sfm7zNLZrgpusWtkno50cqC2x1gIRxw5otQeiixvJ8//bct\naxVh3KBJx1lwrBVMkmou9DTW2/Jl50QtScq3IwDHBhL1qtKD4cFKRHYJDGdc\nJrjY\r\n=lP0o\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-dispatcher-4.14.1.tgz","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.13","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.14.1"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.14.1_1625769631668_0.031864682392240384","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210712.eb38bc0":{"name":"@microsoft/bf-dispatcher","version":"4.15.0-dev.20210712.eb38bc0","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.15.0-dev.20210712.eb38bc0","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"60bbdb862a395e525c42696a4bf7c5c266bf11c4","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.15.0-dev.20210712.eb38bc0.tgz","fileCount":587,"integrity":"sha512-FGh36G4uP2wNrF0xfaxYaEqCY/BZV5cHyhZVpPbiXgxyFWj61SXnkpu25Nh7aXzN4bk9VrNigeQK3lbB27fing==","signatures":[{"sig":"MEYCIQCn0YX62fapfhPkF3mtyU1Yd8j8pDaTvBCQHi6x/nQ2MgIhAKo19C+mkAGi+PCfCA5WFywhzyQZ/EiMPVpNFwHUJKgk","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg6+zyCRA9TVsSAnZWagAAfxMP/jq7Aw0NHFwNIeDTmyFm\ndGr/VXyUJ5AYCFoT7hoeN2EslhnI6c2oLbPfCJrnnOnX3qlFYK5gHx1b7Hqr\n09QFfEthity2EinpX15TZ/8G+3o9rH9/kOiQ5nytOD3UatxKEU0sksKmrkEI\nyYfwMihWuwvPBPFrZr80Pd1bILKjp8Auag7WIAYFu7k6BK7ZM0a6XCX1g8gu\nBQbbV1T5puswiMaX4LWfi/+YV7FESf0cty17tDX7u5lTlY/crmOlXJ8KJJb1\ntIxPtbDyV2xijYu3rKcvBDRHjgI1Ys3cpfue5zpIo1gUqYWvaXtm/PNyNgoR\nmOo58dz83XE4JzrGpEDKX8EvDnJsrIcManUJQ8z9UVo9LohXQyG1dx/5Miuf\nt5/eBTFv4+w7PEvh/f+HKpailqr4mG6sHzKup2+nOOMTVyFcYA0AbdhOP8uJ\nJjdd0coVi9IYoqQ5j1BTkJ3U9wTpsvd/aAvz34ezr6QNMv8HmbdUWCgpGpsG\n6BAzDSUWwsgQM17goZTLzsvppKXlxkK7KPQ/TFpPuuCDnCkgaS2gDUqpfFvZ\n/7exdMEtKMsCxn3r3uKoVGGHZFD5Tz4zD+A6/hHXY3FpvILjETkMERWsP3v7\nRXhLZThBiMoarFz8ndyPcSBr4vaiy8nodL+irWD7l1FGqoSp9dN2aZTCpkU7\nKQ0h\r\n=cG/e\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.15.0-dev.20210712.eb38bc0.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.13","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.15.0-dev.20210712.eb38bc0"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.15.0-dev.20210712.eb38bc0_1626074353764_0.4679945118603579","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210715.ead1751":{"name":"@microsoft/bf-dispatcher","version":"4.15.0-dev.20210715.ead1751","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.15.0-dev.20210715.ead1751","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"39e0fed9a07d112d4e9baf77591b6f5772334446","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.15.0-dev.20210715.ead1751.tgz","fileCount":587,"integrity":"sha512-OtP7j35OcI6ckZHNSrb0bD95Lu4n3/PsfGiLaTiK8pvFH/V5eSPdEsP+LPPnFUHMnL9D3uhofiwa4Vu9mp5aZg==","signatures":[{"sig":"MEUCIQCGFyJxuJz7Q6pd21XURWJ15rCH7zCpzRwrkDrJAN6KeQIgUJIFDkYDPFoToiTf8YeX4/yR6Bx7SwW/on/slfnm1Rc=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg7+F2CRA9TVsSAnZWagAA5XAP/24ilFQN6f4hmfPPrBEY\n/mzgt8rZzy9mcWFSRZab5blq8uSA/3DQNQG7haiOeRxgT6LyogW14gHB6oRn\nyBZzfzt/mTyhSownyK/bgVtj6mpptr7GYYphgx9/FI7YtIo2bCPmV92CKjoh\nMfPYqjJfCUIVQznWnb11mu72eHZUZVukIwiBxki5KHEbXA0CDucjdM1BVdt6\nckdaV5c4rulub3xCrndxBfhR5NMGKK3cYK5PRLHRkJYR3bCHkouasuKdocSx\nLxB1vdb5eETFkqb8mXbv/xHm6gOmWcxf1jrc1qFW2cXkMBjmyEvYFNz9M39M\nJ/+CQ7JBdxwpD9FCzFpAIfGcb2OFmZhfmVmb7cjCUf4tLTHH9MC3lAoch/Y0\nJ4C1ESLe3jp6KrZIld9yZ1BC20R5Kg8YZkf/TIkA/YIk0/bizOOFYhYFschs\nHYWtbp2pGBLjBn0Sr0K0TkKsOQNC6p50+X+rtooS5VdwkmfVJsTgOdOcXrN4\niM0RkBzs2NWZBky6G/HOEtjyUytNeEmFSwVLbCi5tl6AQb33302Kgc+Ow9mE\nu5NbJJfFu6ciB9kGN45u38kR+67tNGRgqa04GVm6UGnOOa839O4Lpg2M8qzY\nzqdy39nABDy43Rm0VmdfH38/W+PiS87ATy0mq+sEUTe4rRpOgycHNK+mcmlr\nhJY4\r\n=mko6\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.15.0-dev.20210715.ead1751.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.13","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.15.0-dev.20210715.ead1751"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.15.0-dev.20210715.ead1751_1626333558419_0.37900707916276377","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210723.bf1f054":{"name":"@microsoft/bf-dispatcher","version":"4.15.0-dev.20210723.bf1f054","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.15.0-dev.20210723.bf1f054","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"e5bd2278a36931b449c238c35dba3bfcf1ed1b3b","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.15.0-dev.20210723.bf1f054.tgz","fileCount":587,"integrity":"sha512-JlDZcSwyOcAH1CbTYy0S4EhYMeU5cBIWNmehsR6Ql8vMbE1Aqq46QOAVn1Lhg4O8mM3zBPObqUrewOSUbGQ9rg==","signatures":[{"sig":"MEQCIEuadJpCcL1oXmWGHk6frKHOoPmchJ+I7I/SL0KVLfewAiAjDNeAO2Typ9RW4d2FDDFUTdgutyE/LuEpS6JogU2CvA==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg+m58CRA9TVsSAnZWagAAdBgP+gKqGPNRCK2LStNi1G/K\nlJHJ1+nJkmjq2/RRQNiohcZertd2vSjJ1TTVXC/7V0TJ0lVQ5jhRWEQ9Nl3d\nbuR1vzjnsLmbp10/lgaGh1nqMKW8VKvPcIGYVIHgNKvfToEd6T2+4f/lrE7v\nUMyQGtyu1W1ZwpZdqhgMuCFP9YaJU8rg0ABn4WrhiZ0zQFgM3gGPSJQjzUz7\nsaNa7yRW2OiyBSWcRlnK/IDS5oWVT7lv3H80O84lRU+mLw2gnbY7VXmS4Xnq\nsHBqas6srnxnAv9mddTXpemeArtMgszBl39axOGTn3EMwjhWR7J1wZeRHOBv\nfVXVVA2cwKmxJa+qm+0qz26toJKFOM9+0yUqxIzcsBI1VEqFDrU7PImxTFsB\nWgKLgqlayyZ+/hA9CU2UFOSKSOwnXmw+2w41EEbvZR5j9X1gpU8Lh7PaP04p\n2uf0qOJ8xbBfeCReAKwALUXf2otlUFlAOVIUWMJEwRnEJIlRRdtpWeNzi1Ck\ngVgVj9M+mC8annOO+H5FM0664dA6N2TSOkQ33c6pWWSt+D24q1kzEraMLyt3\nyqBNLYK75hEqxrQ1kc1AhWbO5NE7G07dJLZONUprblE0Th9LfK1kNiPhS0Z2\n49IK0RCUvnuBddGOUDnQQ8DJq3QgsiDqRdsfRwVU0oZ0pZpiVCSxQlkp5u3e\niviI\r\n=H3Ct\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.15.0-dev.20210723.bf1f054.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.13","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.3","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.15.0-dev.20210723.bf1f054"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.15.0-dev.20210723.bf1f054_1627025020180_0.7279580263502672","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210803.40549ca":{"name":"@microsoft/bf-dispatcher","version":"4.15.0-dev.20210803.40549ca","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.15.0-dev.20210803.40549ca","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"f8c2c41a8372d6d7af0e71c087f207f1464a865a","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.15.0-dev.20210803.40549ca.tgz","fileCount":587,"integrity":"sha512-BDqk/GRoU+UMbbkx8d8XnTtXUdELI8cR03f/LaA1DSw0MRlLy/Lf6EGtkNQpOOYoXT6dfgFc+WX5s2q6ksnoNQ==","signatures":[{"sig":"MEQCIFnYaEWP5LZje8xH4eqqfpT38RfaOc3AqFayt5QmL8svAiBESN2qK8oFRMlahK+h5FYo56793i9YV3p6d0ZtxlGcNA==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJhCO6ICRA9TVsSAnZWagAAo9UP/0JXUBeWAvPpgURyZs/a\nnh5YckJcaIPDRBdKyHNhUymrjRyZlhjldLE/wkjNnlQrmcbmFs28QzeUx5/k\nmDxYBxFeMyH8timt4oNJL0408LLg4t7dIp0uMpHn3Nrt8P8mUvZ4moce51iU\nWcfn/biQ9crN6niKZhHaKJ9lMQCGfB3ikjEr2fkbNRYBIceXlw6ZyGbIQACJ\nUkzul3fNVImAUfBt76vmiSS7PEus8r7PtUon4xfPJU+X1wPNdW9vAR+zy9s5\n6WkLZ+7Hy1YajPnEz37YsH0nOI7T99mKKp2aHtBWzxFWtqjeVlcE39v3zhsd\nx7jAFuUjE01N5+SkE/NqaZYfm9dA3ZbrOAyBQ/mLs84C/6jIUdBZ1g26Pn0B\nkfPjeGlgnf8H2geHpqL2czlVlrPpp4f39ScbYWsGqz2eIs4g5cO1Us099Z4+\n/00kdT86GE0rF9KHn8DS6aSnM4aQCb/HFlEdRirF2geROxjRmJIPb1BnDHzZ\nEM7V6pvAJdu2i+Z6bnL42xvieDbTdJ36khm1kaonruI8cGtz4/NRA3lXtClw\nzW6V4MoLTJlOpqDH5S1sriYL6cf2oWwL8Me8GHYXOm/YZMH87cMWci9xJzZP\ndWLUSzT+kYo5Dq2GEH2tXQ13aLtbEgaho2JozCSsoSdsRR06QwBiBlZiHLUC\nfE6T\r\n=j9DI\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.15.0-dev.20210803.40549ca.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.13","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.3","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.15.0-dev.20210803.40549ca"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.15.0-dev.20210803.40549ca_1627975304355_0.7820509903079864","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210805.6321dcc":{"name":"@microsoft/bf-dispatcher","version":"4.15.0-dev.20210805.6321dcc","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.15.0-dev.20210805.6321dcc","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"6137f039b7ac684ecdc25db47144fa2aec7d6650","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.15.0-dev.20210805.6321dcc.tgz","fileCount":587,"integrity":"sha512-id+p8j2SLp/mxkbeuAtXLgHSBtd4ODLcW1ez7eoHpJyH7+H3CwHQJHrdB9vNKrRBG3Aq/7zI08K5nPmmQRE+2A==","signatures":[{"sig":"MEYCIQCFr91ZXEvIMPz8BSqOcHV02BuMVK00MVG6+N8z0RR2FgIhAJCNWJWyeXAViKaJKJjgH5qoQugKY4yJUj3LW63Yagid","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJhC5EuCRA9TVsSAnZWagAA/vEP/2whq/jo5OEMHoC0wGBW\nRphC6hAb6tT3jtrLiqR9e11UVGhgg4XORt/i/oKSqJgCcEDW320plOTJ3nAs\nre5o+0vK91CnnXdQims4rK1jEvpNr2PFD/si2xmOcgRwrUZt2Ix46wT+2M9J\nmhy9AhAgi3GDOEesCM3qJmfRY8DRZuQxI0La2sRAmP5lQmRMm04xg7mivABI\nYMuwZZ8J/YjgqYC81KCv5UQnl8kdQWOhEzkSI8so25DaCh39j/F2Hjv5mtSI\nyjNO5Vh/rCm9Zn2gvjwyWyIgYvhLs3etuuVj7rLKYvVLmqeuCHgaPZ4b3BG2\nZ3BmuAy+bPRYC1U71lbiTYqoLT0GeS9ve1THAFwLDgmKdoLoDi3b6QUyV2qO\nyeR3z9ihRgVAYBP0P7KCwnVCpkruB13sLn+tRKa5OpTE5QnauZshMte9OHE8\n7AYsCBozzzWrsJdhTdPIOME9Tz2/nfLRuRw28alRrdkjol3p0DpnvYheCxoG\nu6Eh2cSAz2PkXMPqH3w2+HY2wAqOchbp/q+pmijLuULbLMrL+BXe7ASVlayr\nOzwecBKVCCz18JoOmaf5y9lf9l37XXB6FSAi3oe+8e+kWCcDQ0Ansd/hk/LS\nJCsIfiOBLEDws2Ee8TttiGSZLwecoQvgQTNArvbTCsJ6VQHRIzS93ZiX0B9M\nfP+E\r\n=+Htt\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.15.0-dev.20210805.6321dcc.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.14","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.4","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.15.0-dev.20210805.6321dcc"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.15.0-dev.20210805.6321dcc_1628148014195_0.21020405834565992","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210914.7a936c1":{"name":"@microsoft/bf-dispatcher","version":"4.15.0-dev.20210914.7a936c1","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.15.0-dev.20210914.7a936c1","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"f5b1531834ad88965fccf7b78ae818118641fee3","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.15.0-dev.20210914.7a936c1.tgz","fileCount":587,"integrity":"sha512-8oZqzr65GPWn9iYpENLB4cOfnJ685Au4W7V8QIHoFn2C4lj5cNDUp7F6yoIzd0T50zLHbVyf7FeowCuC7yzL2Q==","signatures":[{"sig":"MEUCIQCZKP3vL5PsszULTijnE1cPVwuLr/LYHI3+jmOOeVe58QIgNdJH5x0IgrTWPyYFTTHso510TxWWvZ5VqmGtoCr35DA=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJhQE1VCRA9TVsSAnZWagAAWxoP/2SNnLbRrIB7o16eN3oR\n+GbqBqWYVki4DsXUiiH9QGxaKZ2MFiJn5rt/Lx/yZk+Is+AzKHn7GxudsXmX\ncQ/maMQEep7dDs4VznvLA+394FI3/6HIxOb3kmyqrSohYnAe6Ypf0mOAqL5d\nJEkf8RYndzajLI1urgjw3DgqC2A/dWqRZPIU1WIau113DdLaWB0tfbOqrVbA\nD/kH6Ne01NzEjInttz2RAhAa9xiIoZKj3vJd6ZgPuJaHWsbcsdflau5g3Lb7\nhr3IWikMr/SWTm8JK/Sr5GcTRCznrW6tVKIBp1TAEEr/Joew83KIBF8JI/xW\ncMTNJ63Y3iHEHBaTbWN8Ig2jmKRANV8/H4gnNyGP5mFVoYFX1WewKB3sk3xy\nJVxOBt6k2sSOE11MQcSDngP/X1Lpcjlu2IjBendVnXj0Z0p1WXzIxYLwno5S\nhua1FfXUHbbIqoj3i4isk4coUvNWqlJtJEaNOxxn0W2LGx1N/YMcsMn5dR6K\ne0oqJRdB5ImBQcVg9tPSVgmm/wrbVS+Po7j2L8fWk5TGGL4Iy0qg9cL1Bie9\ndp5duEu+dV7HN2h61/oEEm7ZcuoDhj272N6AwLSyLMn3i3CNBs0ijnmXbk2E\nGcauV/B8tZQ37cDPCc9fy0wPHTLNXjQLIGqkoXAD5jHx25X133B2c6NKZBPy\nYG6Z\r\n=RtRm\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.15.0-dev.20210914.7a936c1.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.15","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.6","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.15.0-dev.20210914.7a936c1"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.15.0-dev.20210914.7a936c1_1631604052842_0.799378191646597","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210917.b688652":{"name":"@microsoft/bf-dispatcher","version":"4.15.0-dev.20210917.b688652","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.15.0-dev.20210917.b688652","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"df4151dd06c48a963ba98c32a708dd565a876159","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.15.0-dev.20210917.b688652.tgz","fileCount":587,"integrity":"sha512-142PcGS+Y68POTkHKvNIE6pZJDKqFicQ17q31zc2WoD5aqv6Cv0jYdjxc6ZUHQyXx8y9f6WjseBoJSJtfEUSBw==","signatures":[{"sig":"MEUCIC+PDawKx8KT0H1o1/6NISvygAhm0rMDFZ1q67xM/yzRAiEA26hG4yaDew2o0/8t97xFJ1qsyUzDqgpyDDNIA5P+ZHA=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJhREJ7CRA9TVsSAnZWagAAE1sP/A7hfT+0f87UYuowVfQg\nZY8eXxVRjbTGdAs/YJcX8peIEa8PIs+F7pVg+Eco2ueAChQWcbIsxLUdZx7b\nz1XFHwelzCK6uSDo0iITyEYsXYv0ssBdI7Y5IK/WGVpq3ot5ArCpm4d6i1sh\nstumKaal4Z9h0NyvUWGauOZn/RVmew7lVYMvZvgfPsH5pRXq7D0GR7LpqPsa\nj2SCm+38a59s9Cr1Hz542nteWhop8Eg1legBJW1ildD6u5I8H9cnT3OFVXiN\n796cwMVjf5CtrOYw3tXYSg+SiZ66CLDj/NPk0VI6gEAYsZOpu5ntgIw31+n3\nyTcjzcYIIes2Q7Itag4bfVFTnvGoKzNQvb3l/CC1Ae1oq8Xizg9OxpODGH4E\nXaHMQG543WTabagdVFjW6d+uAL2ikyAse1JMcPtWzkf/Dh1nbO1vaBxRtfKe\nW/OHtPG8BAZytszmdn8SDmE1Jr83vpktvFNZf2Cl7KWnN7Ww5LqSGDqXuDeD\n1NFXuWDJ/oR49ES/MuSG4TcpbnYON2Vd4C/59FTQbNxxXFem9CcTCSRDZlMp\nNswH3l9yk6SdEbQ/ozEPu5PpIJ2h5BXHxnphu+OY2WzlNiCk/txseDSZ5435\nT9PFMdPwBhzn1G3Kevm/yhHNvRVVDlcFZMJu5vrMHbu7ccFZmctnzzT2MfvP\nexdY\r\n=jPkt\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.15.0-dev.20210917.b688652.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.15","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.6","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.15.0-dev.20210917.b688652"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.15.0-dev.20210917.b688652_1631863419494_0.2921174601781571","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210920.e4d38fc":{"name":"@microsoft/bf-dispatcher","version":"4.15.0-dev.20210920.e4d38fc","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.15.0-dev.20210920.e4d38fc","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"c8a6f88e1e0b9b896c0689188ad051cacd9346f6","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.15.0-dev.20210920.e4d38fc.tgz","fileCount":587,"integrity":"sha512-SQSLNfMfABCVrSoKJtFElHVtu3tFmFiB2SJe/EFKpnecqjDjLTCVSq0uDOHy6eoPcLkdxNYCJ3bwx+2+o1fszQ==","signatures":[{"sig":"MEUCIQDTJquH4FWWbA9avqRaOLpzAtdZEYw5cR+SlkYAlJCQSgIgaLvWghJhQnmU/Qtq7ICcv2vd4+R1kReDnBpwVqxbOcw=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.15.0-dev.20210920.e4d38fc.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.15","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.6","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.15.0-dev.20210920.e4d38fc"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.15.0-dev.20210920.e4d38fc_1632122637889_0.1677839178155336","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210921.fcb2cdb":{"name":"@microsoft/bf-dispatcher","version":"4.15.0-dev.20210921.fcb2cdb","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.15.0-dev.20210921.fcb2cdb","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"6bd690ef22b56d2188e25c02009821ef6ed3f5c0","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.15.0-dev.20210921.fcb2cdb.tgz","fileCount":587,"integrity":"sha512-5fRoDrzBX/YgjBcF8wIEoOA3g9iNiAbONpOcZMpIKzDTOvCSdboKPcNNOvmZ6wV2PEQFv8/E4GEnW0Q5o12EmA==","signatures":[{"sig":"MEUCIBLinFdkeWqhBMKwR4gafUZyxcpb3QPoQZt1Fh2+peamAiEA3wcbJnqklqzJY1tJvL5dtOO6VeYXeaOp1xU9OetKPvM=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.15.0-dev.20210921.fcb2cdb.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.15","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.6","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.15.0-dev.20210921.fcb2cdb"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.15.0-dev.20210921.fcb2cdb_1632208969223_0.8876403283573473","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210922.c1c0528":{"name":"@microsoft/bf-dispatcher","version":"4.15.0-dev.20210922.c1c0528","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.15.0-dev.20210922.c1c0528","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"892ac16bf5f24de8db1a0d3ff22b7de650b60b0a","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.15.0-dev.20210922.c1c0528.tgz","fileCount":587,"integrity":"sha512-TrXUBSHFfg+CvGFe20IVL7scgu5DgNVqJzMxoOo86K5Ct1ctfjPreN+E7jMfcbUa9lYoYtIePZMuzKc9jpCUSQ==","signatures":[{"sig":"MEUCIQDy2xODNY1Tc2wdbvQfEEicgDE4hpWEcpKn/C9PEP0VXwIgbUDTou2IRSg3HLBCPuIwWeUdEsnpTZGKQ9aRPVuNJ9U=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.15.0-dev.20210922.c1c0528.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.15","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.6","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.15.0-dev.20210922.c1c0528"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.15.0-dev.20210922.c1c0528_1632295324214_0.4805025330029755","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210924.20a3f98":{"name":"@microsoft/bf-dispatcher","version":"4.15.0-dev.20210924.20a3f98","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.15.0-dev.20210924.20a3f98","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"2881a0c0a4810faf4d69172dd003763b2e11bfe2","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.15.0-dev.20210924.20a3f98.tgz","fileCount":587,"integrity":"sha512-kBShTfVmX+dbqLFUptDQtIGp7tiqRNsksbwBTSXjejPY9nQMlYo4gz0CfaPHu3YJh7QnlKRVjGzYLQax1aSvpw==","signatures":[{"sig":"MEUCIQC62Y41f2fQGLO6m9yH82Z/h2SJILG1sbWMgMAGhtQX4AIgdujXqOmNT5zCiOQs5+bzIYbq9VC+4wULMj+fxh4UJX0=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.15.0-dev.20210924.20a3f98.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.15","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.6","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.15.0-dev.20210924.20a3f98"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.15.0-dev.20210924.20a3f98_1632467951702_0.22243072161560318","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210928.a9a92a5":{"name":"@microsoft/bf-dispatcher","version":"4.15.0-dev.20210928.a9a92a5","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.15.0-dev.20210928.a9a92a5","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"7c7897b8d097d05b0476b902debf8d89ddef801e","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.15.0-dev.20210928.a9a92a5.tgz","fileCount":587,"integrity":"sha512-WGhWcNCiwX38RaRHlzcigNBynYHpwOSsgJswUAQcNWsdewr4rxO1MPanX8GMlbZS7Iy0cFAjCQEfVRL7+HifXw==","signatures":[{"sig":"MEQCIBOC9lvhoqP1CvH2bSa4GXKj1W67uewnF4M5oZ+Idh0BAiBND5pFxV1Sm/d52aZd1CGPk3HZa+EscdViE2/zDTJK+A==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.15.0-dev.20210928.a9a92a5.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.15","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.6","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.15.0-dev.20210928.a9a92a5"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.15.0-dev.20210928.a9a92a5_1632813759284_0.9813256911809611","host":"s3://npm-registry-packages"}},"4.15.0-dev.20211004.edcf63e":{"name":"@microsoft/bf-dispatcher","version":"4.15.0-dev.20211004.edcf63e","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.15.0-dev.20211004.edcf63e","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"0b921d1156921414d950e596eb510d0f2a4ea29a","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.15.0-dev.20211004.edcf63e.tgz","fileCount":587,"integrity":"sha512-75imoQSxyNrDepEsqmRufWBx5Rp1mIC1Dl6FohvmHLWmkuOgbB828sJ3vMJEEoUSNwKB5u1P8ml4FqJHBD47fg==","signatures":[{"sig":"MEYCIQDtO/shqKNDUxq/W7sQz4oFbBoiDYMYpSkciwdw19XCwgIhAPGjN3WL2i0N86TRSMfEwCbBYPuATfsJU2vy25gdVhsl","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.15.0-dev.20211004.edcf63e.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.15","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.17.6","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.15.0-dev.20211004.edcf63e"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.15.0-dev.20211004.edcf63e_1633332304996_0.42020411852322614","host":"s3://npm-registry-packages"}},"4.15.0-rc0":{"name":"@microsoft/bf-dispatcher","version":"4.15.0-rc0","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.15.0-rc0","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"c689284459fdd3c9a69a4162143a523b57a800a4","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.15.0-rc0.tgz","fileCount":587,"integrity":"sha512-eMqbVv9fmCK0oPPDOVVHzg4FS4lQQWqt8laILWiFOkJTlhwxfA7/H1Qjohu2s1EnjklmXYPSkqmy46P9GACSBg==","signatures":[{"sig":"MEYCIQCOfYMRK2RrzahKXHG/fSX6LqV59z4IoPZyvrZGPB8p8wIhAKoV5gtp/Pi7IgBuwRUyduiWcdPeHE/YhetLJryll0d2","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198639},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-dispatcher-4.15.0-rc0.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.15","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.18.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.15.0-rc0"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.15.0-rc0_1634338333351_0.8386097224868194","host":"s3://npm-registry-packages"}},"4.16.0-dev.20211020.406dfef":{"name":"@microsoft/bf-dispatcher","version":"4.16.0-dev.20211020.406dfef","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.16.0-dev.20211020.406dfef","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"949490641eed66801c68a61bcb98d52cdc18d41b","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.16.0-dev.20211020.406dfef.tgz","fileCount":587,"integrity":"sha512-2QGv4OJXXiXyV84u2B5XrlknemZnJ0BNB5DYoKZ1r83a6vAYNtqLDGdYy6+xiiNZcRzMQql4Hw0xddm4LvTWjQ==","signatures":[{"sig":"MEYCIQCBdiM5cZTXfqvwuSLQBE2H5zfYqLgAmm+FBRG6ChfprgIhAJuTMgMC1aUcQwZhtevALtdXyJ/uRFTweLC+gR4VS8qE","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.16.0-dev.20211020.406dfef.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.15","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.18.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.16.0-dev.20211020.406dfef"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.16.0-dev.20211020.406dfef_1634714621523_0.1952389828423713","host":"s3://npm-registry-packages"}},"4.16.0-dev.20211101.bed3c05":{"name":"@microsoft/bf-dispatcher","version":"4.16.0-dev.20211101.bed3c05","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.16.0-dev.20211101.bed3c05","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"d3d60313c75351823c0831e41a77b13581b79bd6","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.16.0-dev.20211101.bed3c05.tgz","fileCount":587,"integrity":"sha512-51z5UcJ81D/R+aEFevYu00gBUSRIGC6twj134R0hDBcH4pYHZd7TPJ4fil6XFDBHmlO7WuQqwQz7FpjXQpAi6A==","signatures":[{"sig":"MEUCIQD8rddqPZUDbvI4nmlmojXTLWNtalIQ0awQb5p5cB24HQIgeM1SHQAcHhDpD1FPmlnBVBofbaibWOUcUFAb0Ly1lnY=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.16.0-dev.20211101.bed3c05.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.15","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.18.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.16.0-dev.20211101.bed3c05"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.16.0-dev.20211101.bed3c05_1635788127827_0.5381994942839796","host":"s3://npm-registry-packages"}},"4.16.0-dev.20211102.bed3c05":{"name":"@microsoft/bf-dispatcher","version":"4.16.0-dev.20211102.bed3c05","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.16.0-dev.20211102.bed3c05","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"f9c09693c25490966bb3d8a52b5d034f975f282c","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.16.0-dev.20211102.bed3c05.tgz","fileCount":587,"integrity":"sha512-YUEi8PKgxe0grAa4Jn1GQRKfepfNA6Vvr6hg7H12LB3s2tEXYxzKdWH+HPmmw9A8uhFeNsv/OnxiSNrgM6aU9w==","signatures":[{"sig":"MEQCIF2WikYE4xYbtn05CFSQN/+SYWWtDxknkUqrKa6iHwE4AiBzdrkqnWjMKA30niT9CLX+eqqFc2Ic6T8qdF+6d6t13w==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.16.0-dev.20211102.bed3c05.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.15","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.18.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.16.0-dev.20211102.bed3c05"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.16.0-dev.20211102.bed3c05_1635837876006_0.7832363652401875","host":"s3://npm-registry-packages"}},"4.16.0-dev.20211102.3f7a67a":{"name":"@microsoft/bf-dispatcher","version":"4.16.0-dev.20211102.3f7a67a","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.16.0-dev.20211102.3f7a67a","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"9ae75834d36972cefca12c33dc9c6e709293e74d","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.16.0-dev.20211102.3f7a67a.tgz","fileCount":587,"integrity":"sha512-nbd9jfBhSagES46CJgAB5qcghW5w6MdIeOtSv8ujO2hPDM+nuSytdrW3PbOArjbOh5N5kmct47j8mHzpnGd1HQ==","signatures":[{"sig":"MEUCIQCGmEYKy67uGM3nQaM/wtGuC8eEWYPbHP+XemjLCDtbTAIgIzmAEhAiQxzuR82PUlL9OiVyfNtUEuJ9iqSj9WKXUhY=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.16.0-dev.20211102.3f7a67a.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.15","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.18.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.16.0-dev.20211102.3f7a67a"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.16.0-dev.20211102.3f7a67a_1635875287750_0.6893460126654121","host":"s3://npm-registry-packages"}},"4.15.0-rc1":{"name":"@microsoft/bf-dispatcher","version":"4.15.0-rc1","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.15.0-rc1","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"042e78aaef0a503dee157eb33e240fa8c6de497c","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.15.0-rc1.tgz","fileCount":587,"integrity":"sha512-nI4w41ItWtx7nJMJY6vQGWw1as1ghoBH51C12mVe6J4LspGfV6ZJShD40e5k77akkfBd57ZpvJi/MbA451Kcxg==","signatures":[{"sig":"MEUCIEprApGG5RwDyKvBHopWgPJOEJ7kUHJoQqNENwmeui73AiEAwYiJnEJedYhCvWJPVKKAnpKppYkZ2IPC6/BguN3W4Rk=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198639},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-dispatcher-4.15.0-rc1.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.15","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.18.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.15.0-rc1"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.15.0-rc1_1635895039300_0.708885878021972","host":"s3://npm-registry-packages"}},"4.16.0-dev.20211103.3f7a67a":{"name":"@microsoft/bf-dispatcher","version":"4.16.0-dev.20211103.3f7a67a","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.16.0-dev.20211103.3f7a67a","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"3c48f42e1fd6bf5c0ad68b13daddeef8f4c72e08","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.16.0-dev.20211103.3f7a67a.tgz","fileCount":587,"integrity":"sha512-KVxCY8BQMYQPp1RPcc7Om3UKM/g1b7fe5Vg3ZIBZ+35O+KqTeTW1DtrlioKWprgdLQHgyKYkgj1Ja9KLXdzVmg==","signatures":[{"sig":"MEUCIGswmNwu7Q11OWdByfABNJPdEL45OA9lSqaY6x9JRB/bAiEA46ZAIFO+WBW0GWVxqbO1P0NaSS3LbG0S/kkpTm1DoNg=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.16.0-dev.20211103.3f7a67a.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.15","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.18.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.16.0-dev.20211103.3f7a67a"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.16.0-dev.20211103.3f7a67a_1635924011988_0.9822292289864101","host":"s3://npm-registry-packages"}},"4.16.0-dev.20211105.90655c4":{"name":"@microsoft/bf-dispatcher","version":"4.16.0-dev.20211105.90655c4","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.16.0-dev.20211105.90655c4","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"11d433692a6da070f4fe211aba20d9befd0e77cc","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.16.0-dev.20211105.90655c4.tgz","fileCount":587,"integrity":"sha512-MnqJpdahGZFJ0JuWHOmP6QYocaAYl3Fhck5xiP3a7/YjTOb0//z0ZdhC98XVdcsn5JO+SAdRokJmcbqZh5krUw==","signatures":[{"sig":"MEYCIQCv11YoB+JLnoSfenmW8/KebRC4r7vwAiL3oyzR1GuWBAIhAJZq/0P/Q2YN5Hgt9xZWIIB2VKpKJWMqBmDnFYj7FfYD","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.16.0-dev.20211105.90655c4.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.15","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.18.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.16.0-dev.20211105.90655c4"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.16.0-dev.20211105.90655c4_1636097086194_0.8813100030256447","host":"s3://npm-registry-packages"}},"4.15.0-rc2":{"name":"@microsoft/bf-dispatcher","version":"4.15.0-rc2","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.15.0-rc2","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"053afba839a01277f11f564479cc053eb7611052","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.15.0-rc2.tgz","fileCount":587,"integrity":"sha512-mYGO90iisZeDT90li2Hz2FZWM2C0r+HG6afFxBQtjBVMEzoP64OaTTuIsni3gghpL/TZAZiPv+BcosOrqdz6dA==","signatures":[{"sig":"MEUCIQDsib1OsKXADeW2nXEZRS+leBahB4VQ/GsKWJL6V30nJAIgJvZqWvgVyhYRwDqMav2vOfPXwA/Yiy7bSrvHNCjXxC4=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198639},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-dispatcher-4.15.0-rc2.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.15","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.18.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.15.0-rc2"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.15.0-rc2_1636136479733_0.5873045633481868","host":"s3://npm-registry-packages"}},"4.15.0-rc3":{"name":"@microsoft/bf-dispatcher","version":"4.15.0-rc3","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.15.0-rc3","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"5aa8258e3eb9cc50d2e968ffa66b9c581432d53b","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.15.0-rc3.tgz","fileCount":587,"integrity":"sha512-C4SnwzXno9ASa5AZYmfU/mfdnoOqKPaIqe/8zcQkw1RZ6FtbbyjOUTjZNpCCmCpKCrMgbgr2wLk+Q6nZp9pMJQ==","signatures":[{"sig":"MEUCIGR3gyQg3hMcqnaOR2xW09U6xhfa/R+vNPy5Zcq+98xVAiEAqiSDCf6fmq4C1klRNloBzqxxud0xa9txq2jr1m3PZO0=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198639},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-dispatcher-4.15.0-rc3.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.15","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.18.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.15.0-rc3"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.15.0-rc3_1636587614940_0.44076704257047394","host":"s3://npm-registry-packages"}},"4.16.0-dev.20211111.15b5c37":{"name":"@microsoft/bf-dispatcher","version":"4.16.0-dev.20211111.15b5c37","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.16.0-dev.20211111.15b5c37","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"3a68a14fcbc9c5e3291e57ee946d22bb761552bf","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.16.0-dev.20211111.15b5c37.tgz","fileCount":587,"integrity":"sha512-ese7n5K/G7zKRHqUkciZUlKt4RClrQSVNyR+46Bk39nLWIedDCLic6zEi3/c5Aj8HeTQ+s1sZTaJO/a16K+Vkg==","signatures":[{"sig":"MEUCIBf0XZ5FY3MflgEY/mQvPj5VcGA113LSKCgZxTS2zMXXAiEAsZx6O7sEvsEvC6xzCKciFLUYN7FRxeMAfACQNO6rdQw=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.16.0-dev.20211111.15b5c37.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.15","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.18.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.16.0-dev.20211111.15b5c37"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.16.0-dev.20211111.15b5c37_1636618939585_0.876257834913668","host":"s3://npm-registry-packages"}},"4.15.0":{"name":"@microsoft/bf-dispatcher","version":"4.15.0","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.15.0","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"0661ad97d7615659ff2ba47411f9a755569e94e6","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.15.0.tgz","fileCount":587,"integrity":"sha512-YlCV2IU6yeTWgTHw5JUnDYurtAEdIedvqA+sdtWM0JjkhtNulZLAbZ4k+EGr94yLe4fNXHPqSg9FLXTyeBUgrQ==","signatures":[{"sig":"MEYCIQCYlD/KUJCrI6lc0l01BYYco3zUGrdjCSVT76aw+DXT8AIhANzXzPZPhbgV3YUwuF2TsexXiKyl5Bp4aZ+SADgIjVbi","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198631,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJhk+Y5CRA9TVsSAnZWagAAC8cP/A2ysfj3rU352q12A8pt\nUSN7Q5Z99v0OwKkcsm8w8Nk1mZwFOW0OY6hOsxtGhv/HTI6e8umhh1pLKBAM\nymOgQ2uBjdwerIWlPIUmUO/2Kc+p/B+n8b78wu3ZT3piGXNTv5pEipCm5dCU\nzFkbojJilw373LA3E0FpI+J85HcMc1hGvZTrqAzqHcEeUARA3quKJxUek07p\nb25EL9Yvi9MY625bHaN540w+8k0Ghu8+DlMP3Wm3AHbI+cGErpsHNI9zGW+v\n59/MvNOEwSERF555bRK9cVJVXNX2r8RX8PXG7W7oS+2gmaTchpfVNnUrBY11\nI9dbdM+eUILnCfFr8S+hHj/RXFWcc3LnRNOHTR4uBVkeSGJ9AD35iYRtN1At\nHQmSnS2RHqk140XT+y8GbrYlbQD43B4ufI3y2IVWXODPrLkmT8Wx4/ef1Xp/\neCQG43wrMO474LJ/st6PPe64p9Qbo2MkYLAHB2WJjc227WggqQy/4frLQZRp\nnF4pLfq9PxwGwaDjKiowdwW3iqdiKyr/29wjSH0gafEv/+7/18xRfhZmo429\n/zCj68dNSamkIpXppCso0etcnaUaADestDOLDrKWTzACQ9e3i54xaUnK7L4Q\ndCCuT4fltx5CC0L1yIi5G6uo5mn1mOlr+UElZ2IBmbpVODUgz+oGHNu4U7ON\nSliN\r\n=bBN2\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-dispatcher-4.15.0.tgz","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.15","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.18.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.15.0"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.15.0_1637082681653_0.27755160680894186","host":"s3://npm-registry-packages"}},"4.16.0-dev.20211117.19ff452":{"name":"@microsoft/bf-dispatcher","version":"4.16.0-dev.20211117.19ff452","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.16.0-dev.20211117.19ff452","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"ff002065d81f02f8bf8dcf4329f8d0295c4886e5","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.16.0-dev.20211117.19ff452.tgz","fileCount":587,"integrity":"sha512-13mQKxvUUBxeWUJD9iurte5fRagkDDcBfzaUt9DcPTiVlfiTgYJp/qUj20X6nYzRXm1I2k5lrCu64tanQcuWVA==","signatures":[{"sig":"MEUCIQC2zNgrvOxT43SiM2a7iWtWGq3hR7Qb4dF6o8BGAFVjHAIgHZRxaCGj7V5HmPb6zyDKGFA91elhQjNMBa4vBg/R02I=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJhlLv7CRA9TVsSAnZWagAAlWoP/3vC8G1NEBpuqUaoeU/V\nzUEfda+Eik/4UT+PuPM22M4ZxgmnAm+HWgoUV3jk9zsTlzdVlkOAmnQzxsJi\nzjvf+JJSjoWr301a7UG+OpqW0WMwzvCmda5kYd/iKcqWYFrTsKVR1b0F45Ll\nflQrPO2AuJFDGzgxEb1F9OGIo7TVrYnp9ervcjOZk68Z84TzPGq4sLm/67Q/\nvaHj+VwJsvWrRuLzOkE90DZzPKCe/dtumnp2ZpsztBdZGN8izSQiDzzj4Emo\n0KzjJaYSUJHQqdGHUQ1cLHKoC/L1uO0yBswCdO33N+Lrpm4comCmL9lkv/WU\nnr9JVv7n+LirkZKBP5u5i78GbWVfczi0ia04XT0kdGEDLZhNM/Qb0sgkZKrT\ns1WjNexPlTIJ1zweqNzB9cnrafO8FRBe/MPMlNEN71W9W4i3cF4WL64WNMAI\nerS4XTdGktXAIuze475ymk7pRCaFXd8Gdnw1uQk/QDK80Q1KLBXwQyiF0qj1\n3GRFDygwtgRts6umpE2szWNPHLhRkIncElM3pQDvkGzJrogrOG0v6dKUt601\nk9GiyGQ/mDDt5VWy7Lk/Iu9iiIO77r+IdkbbOJu8+Ixyo7RVTFJP8XYmjBga\n07FK4KVsQyOVvXXkgmHt00y6dsn+3xUSz6BnX0dqNn7/iq4zyCDisafaZwG+\nsG4m\r\n=cQ/g\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-dispatcher-4.16.0-dev.20211117.19ff452.tgz","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"_npmVersion":"6.14.15","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"14.18.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.16.0-dev.20211117.19ff452"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.16.0-dev.20211117.19ff452_1637137403333_0.7199794597034013","host":"s3://npm-registry-packages"}},"4.16.0-dev.20211220.6972973":{"name":"@microsoft/bf-dispatcher","version":"4.16.0-dev.20211220.6972973","keywords":["oclif-plugin"],"author":"Microsoft","license":"MIT","_id":"@microsoft/bf-dispatcher@4.16.0-dev.20211220.6972973","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":"https://github.com/microsoft/botframework-cli/issues","dist":{"shasum":"627d897fd7f0887e1bc78dd1a632cded03a14eba","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.16.0-dev.20211220.6972973.tgz","fileCount":587,"integrity":"sha512-fvmrIK5Va9S3cMCjOXz7JN9daqaCYRcj7OVjCO3ywIOGoqxnO7IvtjUtpJVZCvAjI6L7taUAjyTxlm5YyMwtcA==","signatures":[{"sig":"MEUCIDTGNNm5t09OWSTJZDH8nUE287KTLisM/4VHe2f0TPZwAiEAjNmgcnj152uNRToS9FG7whf1x3uqEUGkX7TUdpVG0tk=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJhwD2BCRA9TVsSAnZWagAADD8P/iu8iTfnVVsZsRjd57+U\nwBLcOWUOnQkSzYOahFIiHCgz7GKY34Irm5F4JrpR39TUheBbBeyMA7soVhXU\nEcIkmptXo3etbBS6NrjxrrryzEkD4nx20cQf//aBjqYQ9hfKvdxBZKBBzDd6\nVGMH1sRTldTXFMgZuFKmTawLi1oje2amMCu91mnOIO3t6cIAaVS/xELOyHDL\n5df6BtuY83aP4pzgPFiv+nzQjGDki/rRXYNUKu17ZpONwsE+IiemE4YqZnRR\nsDpCU50uZGM7lO04Q55pUOKFIsPQ01ElUBJmCpYxQiJVrK7q1RKcWjbuBu+0\nl/dTceszYtZNEAfB97eqeDPR7bf1QBp7+HTlDStdAs6thtwoEZEzrqXZw8WW\nNatX7/x9wTYvv83Jwwupzl/grADqAvVXlk/mMV8mKRAR4AmQBsLS5OELgZYH\nEwgX5vv87dae4MNPXt2Inu4IXT42gd/0WgyUY7EK2FS6u7hsXO0S3oFnzMeN\nyg1eKJumCGvE8uLAqv60c8Fich9M7M5Y88YGGewo7Z2OC611gQmnxlrltXSy\nqu13CJKT75rEfVkjYEZG4OkuyM5hsm1Fr8a8FdaoizkHvFIygCCgGwJL5jbb\nk2snEOXDUJzggXhskkVtLFtDe0AH7qanTsqu3BJxMv79cEwc91Y2XiFWvK6a\naQvA\r\n=ywYN\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20211220.6972973.tgz","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20211220.6972973.tgz","_integrity":"sha512-fvmrIK5Va9S3cMCjOXz7JN9daqaCYRcj7OVjCO3ywIOGoqxnO7IvtjUtpJVZCvAjI6L7taUAjyTxlm5YyMwtcA==","repository":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","_npmVersion":"8.1.2","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"16.13.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.16.0-dev.20211220.6972973"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.16.0-dev.20211220.6972973_1639988609173_0.4584314984531621","host":"s3://npm-registry-packages"}},"4.16.0-dev.20220104.33b5e0a":{"name":"@microsoft/bf-dispatcher","version":"4.16.0-dev.20220104.33b5e0a","keywords":["oclif-plugin"],"author":"Microsoft","license":"MIT","_id":"@microsoft/bf-dispatcher@4.16.0-dev.20220104.33b5e0a","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":"https://github.com/microsoft/botframework-cli/issues","dist":{"shasum":"1d7ef182382dc4a22e12461dd2bc1a90fdecc054","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.16.0-dev.20220104.33b5e0a.tgz","fileCount":587,"integrity":"sha512-giPVbPDLoNQ6/6pWsmDIiEkKdv0PQvP7nAL6dmCtvfOS3kwOscWGuvZZC8obI3SdOOlpe4R+2/mSbR4cXFoWUQ==","signatures":[{"sig":"MEYCIQD2QEdPaptVCAOBBONpwgzE1cfT1aS0qTVfSQVHh/roigIhAObYTmzdun8+Z2J/1j+85dXCg0H3/pjTg5hTqyaEs97K","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJh1APsCRA9TVsSAnZWagAAWp8QAKSdclH8zXHdr6BrT+37\nMLwv1EMnK/VWlZnsU/DIGNMYAdJcUmB+Tbj7BUPwBuEzgVQaORuvXno+ypFS\nGoYLZVlvLm07lsMwpyvqEARmmFcyFDpk9R4I8vjO0dmjy4WD+9iOhz2XP2LR\nsc7EY9HKdYVv0N+iezdCtYlB1nrgUUXGvcJSwbpGEnOdhPibpwbfEMuNTDJH\nMtif3W43e1RyZWVhseXb4k2PVirglGAoXzt5zDvubksjV71W6Po3JRGm5v3Z\nSAie8p8sAOAorJ8oRzcQM9Imk9elcIZ7nQ/vIInB2ENfUPSqLogeobstqgac\nOvZdZsxuhUVNVzO5WwhtXOv3p9ohyte8NJ2/GaCKomDU2/xT5oWHht4z36Q7\nWsLmcqbHUOYbi0pTMtvGTdtWV9TBMUQLjtNL2SzGlZcZFzs5IdSgboQZHzWt\nr1MbuouZMrN9FrNDzJ61WOS7VPYkWBYDEcT/dLw0pu5svddLrokpWtitL0gY\n1uFWlQbnMUfNJ5vxwjawPco/00/40Yw2gXFrCz8xpP4xnF5QEaWleeBnl7sz\n4VcffU3n5fbwK7mOskjHea5W5Hz3g20nFXD1Ly7sqj8HkFfdQBRYN3Tfa6r0\nfkB9dWV+KV7iDXm+Bbw1G7a/WZN36JAp22XfNG4pT9771iiyNjV8Tpw4AJ7l\nKU0+\r\n=ozjV\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20220104.33b5e0a.tgz","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20220104.33b5e0a.tgz","_integrity":"sha512-giPVbPDLoNQ6/6pWsmDIiEkKdv0PQvP7nAL6dmCtvfOS3kwOscWGuvZZC8obI3SdOOlpe4R+2/mSbR4cXFoWUQ==","repository":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","_npmVersion":"8.1.2","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"16.13.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.16.0-dev.20220104.33b5e0a"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.16.0-dev.20220104.33b5e0a_1641284588440_0.30103679077748","host":"s3://npm-registry-packages"}},"4.16.0-dev.20220105.380511c":{"name":"@microsoft/bf-dispatcher","version":"4.16.0-dev.20220105.380511c","keywords":["oclif-plugin"],"author":"Microsoft","license":"MIT","_id":"@microsoft/bf-dispatcher@4.16.0-dev.20220105.380511c","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":"https://github.com/microsoft/botframework-cli/issues","dist":{"shasum":"55a38d11e8ebdc5fe8c01e8f29362f1cd919f3f4","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.16.0-dev.20220105.380511c.tgz","fileCount":587,"integrity":"sha512-plLnPZkQ9kLNpF6aDNVVobIgoGhnIjLu/jFor0zYqqDwM4tHxrcJEiOvdwUwYa9D/YCyp5ubI9ROuUCQosoJ7w==","signatures":[{"sig":"MEUCIQDNZnFS8ZJw7df7hov7LXMFpZEL4rAsZc+J6Gi3TJewiQIgAqDM+1sIrqpKC1b52CVknK0IOmnrerUaKL8dDcQmgUs=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJh1gkaCRA9TVsSAnZWagAAEhsP/1KekmlPx8PpTjLd/a0C\nbr7lV30N/NuOVpn+olHo0lFM8hj5GzO3JnSV6Vb+UmocKHX12LCcsI5wIAZ+\nUqNu7fyHTcVx621ZY1QEIcpXBf4i2uljAhxcC7xdnvbpXIxtBOcGkXYEITO2\nh8+/VpM/doLdKZ+eEwuHeY9UDxsYtBR9lCLRf0/T1sxIR1EamQgifG4ICxEH\n1YtRdyMMuhJ7uZfMwjJenFUyWy+Xv2IwEh+ILW0faq/Vk90L4O2CyJGDYZEu\nXZ/T28gBGhYEnaKrZ9Itn7h0TDVRjaOIx1X1n0DqsjYJCxWA2EZMvgr3nL7X\n1LM44/C8vJVeuuHPRBg983hlE7HLFpUsJ8DuwdWTWKUy7kCPbRDlJ/VV2UoD\n4fqucLP4t0rChgE6BYF1nahFVL+bqrbo+mf/ylDsxmACgfN0g4Z9yIsWhvnD\nBoO8SCjoPw+vhpR6xaDg/3Lp5aPUsZBxrPPPFdRTmQmYDY+j51/gp8UOpv0e\nMgogMpJ708niz+Yw53YsxB7+BRiZUq3F4Fx59QL/2lDRPyAzp+20dpiOlQLD\n6W6F1YDwSf/s7MDAvf80pf11XBO+OesbjZGqgbPKA5+FrTjLJXxJHlahQjPw\nCfP+d/OB+rCA4or7rrg9OHAuhCetWyKfiMVZ0QqE5JWOF/AsJ/rOB3EYoQc+\nSnQ+\r\n=0EcN\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20220105.380511c.tgz","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20220105.380511c.tgz","_integrity":"sha512-plLnPZkQ9kLNpF6aDNVVobIgoGhnIjLu/jFor0zYqqDwM4tHxrcJEiOvdwUwYa9D/YCyp5ubI9ROuUCQosoJ7w==","repository":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","_npmVersion":"8.1.2","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"16.13.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.16.0-dev.20220105.380511c"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.16.0-dev.20220105.380511c_1641416985925_0.8883953153646256","host":"s3://npm-registry-packages"}},"4.16.0-dev.20220105.c44b989":{"name":"@microsoft/bf-dispatcher","version":"4.16.0-dev.20220105.c44b989","keywords":["oclif-plugin"],"author":"Microsoft","license":"MIT","_id":"@microsoft/bf-dispatcher@4.16.0-dev.20220105.c44b989","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":"https://github.com/microsoft/botframework-cli/issues","dist":{"shasum":"ad033ec59a98c4d9e2317008037f8015c57d3b41","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.16.0-dev.20220105.c44b989.tgz","fileCount":587,"integrity":"sha512-KjiSrqEO+ATFRUoKIsN0JlyWQEX9MtMSe6GhF5YW+tyz9D6ePDp9Fax1xFyDJoJchA6+os7cqneoVPRBGuh/SA==","signatures":[{"sig":"MEYCIQC5C0eXIEAf7f90fZmzzv39v8XzJfSPYHXXrQDn4BuLxgIhAMzD1jRFSWZAHHa3dcVirgU/r8vwGlJhnHKuEkbfTWC7","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJh1jFYCRA9TVsSAnZWagAAIT4P/imoRxvPDUii5NPsh2Yp\nWx4fAzsiVRKFK8cKMXih+jlj1NCKNoD7qX/ctK56jvK05DhxkWkGuq6+kWvq\nc88DrRCaIZLMTU34wo6l/crw61xCoHXEczuhQzdAjDPxU91aH7U40X4+4rOZ\nCnV48ZsB1jA3KxAFxz+xHh2kmW6JND0T44ZWOglsHzeSYACqVkdV8XTTlUMi\niddP8rOIvSYkOT60llMYhu8bSROVne91NEi3ktaKJV9mQqV8Pqxx4LJisHBp\n8QvpGsySIIN9GUzt/0yfdGtwOrjG/ipQwlb0uEay1x1rg9OgEzcZgSo5V/E6\n63EXyT2tQ/rQGBBovFMjI28fRskmUJGPTQ2r0Nefpizj/Kgd/bJK+rRKdLwc\nKsWHREl07atyAUeiKn4ZwNzGynw+1dcgkfd3X+vT2eZSGUse58XKGF34UDvv\nhkwxnEPAcX4lxt/2HXzXIRDppdINhExCynbgftkKC/+zaAevf4jQvR8J90d8\nPL/vQpEjfbI6sl7tYadTJItaT0CW69iw4MgXEuR39A7p7brgktLO43nJ1297\n7KCaVOXOkUZpd1WLdabsEkq5OHEB/lGdUTWLaqwx3BmyrhuUxEXx5ZpMe2W6\noUnDwLrSryt7swJuZ09bNGGnn9RqRWZsWPKZb3a8U5/eBCfokLGwJx6dfZ3H\nOFIW\r\n=K3gc\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20220105.c44b989.tgz","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20220105.c44b989.tgz","_integrity":"sha512-KjiSrqEO+ATFRUoKIsN0JlyWQEX9MtMSe6GhF5YW+tyz9D6ePDp9Fax1xFyDJoJchA6+os7cqneoVPRBGuh/SA==","repository":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","_npmVersion":"8.1.2","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"16.13.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.16.0-dev.20220105.c44b989"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.16.0-dev.20220105.c44b989_1641427288254_0.8237936228774383","host":"s3://npm-registry-packages"}},"4.16.0-dev.20220106.c44b989":{"name":"@microsoft/bf-dispatcher","version":"4.16.0-dev.20220106.c44b989","keywords":["oclif-plugin"],"author":"Microsoft","license":"MIT","_id":"@microsoft/bf-dispatcher@4.16.0-dev.20220106.c44b989","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":"https://github.com/microsoft/botframework-cli/issues","dist":{"shasum":"a444aeba138a7f904575c591d154dd13a338b29d","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.16.0-dev.20220106.c44b989.tgz","fileCount":587,"integrity":"sha512-afUf8XFPfz9gH259Yz4uUkON9V/sr4xNW5TLOOarsU9u52TjrLeR6eyWHo6iUvPyfYtXcjnLIP/wIqmyd3VxyQ==","signatures":[{"sig":"MEQCIAtewMegfwH0q4KZxEDt1pNCDOdS9omwcOdVfcp6IUU0AiB3+/l5oDN4DpkNwDQ5zFKLqKKOwCiHY98CoqQzhMdEJg==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJh1qZbCRA9TVsSAnZWagAAJZIP/ikH6jTfrV5V+YkNSaTp\n5pfp7bjPDvNr4byq+WLuVhhx2+cW7TCOCSUz2Bs7LBv3aK4weMYAZExavYv9\nrNNCKDd/QFA0h14ydJeaOJTXwOLmsRYnu5k/AUo0qFZ5fUoh0FnuOeuUKiQV\ns5rKxAJD7T3Yq6vkerPKK7zk61WAgXgKivDEHAqFl+qFTYGsvrGwYET5onAL\nd2W8Dpkbdo0KH+SFwws3XraCRTyER4mcg/aykg9J24P8hC4919hkgkD0C9HH\nCIqgXDqcu2QTs0oN9vhg5T5MYWcT7q+x6SqEXGtnDNqfZdo/71hFZ+inhXCQ\nUBVhRbLdq9Tob1ckkHl8lv5eLGa3N9McdzgOmvrg3b/CT6zjcOePyhDCzXgk\nV/wlQPmt/sy8ttRtpol8uNRMsKFULbJjAYfLY74LyEw5sYxlNbzyuOA5XQPu\n49MGeM6V80wWTugjSvHhApqx/yUIyI+u3YBPUyLajFBH/dV1S5JhMGwpyj0i\npJSKolFFo0tTBAL1e2Dw/COEZaHXJRiWpBxhFrxlDUqzatq15DlvLkkxbSbb\nG/ruom7gTe11pMlLffidUpv44jBtTG3KWyDaQytWcwojNSiBUKiU2DftoVIT\nNIe1chAkttGBnRud6FxfyudVMtZHutXSS5XbLVBBKkpBA+4IxME7tCsX85+p\nl/xF\r\n=sEYM\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20220106.c44b989.tgz","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20220106.c44b989.tgz","_integrity":"sha512-afUf8XFPfz9gH259Yz4uUkON9V/sr4xNW5TLOOarsU9u52TjrLeR6eyWHo6iUvPyfYtXcjnLIP/wIqmyd3VxyQ==","repository":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","_npmVersion":"8.1.2","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"16.13.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.16.0-dev.20220106.c44b989"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.16.0-dev.20220106.c44b989_1641457243032_0.6148508785891087","host":"s3://npm-registry-packages"}},"4.16.0-dev.20220110.b4d497d":{"name":"@microsoft/bf-dispatcher","version":"4.16.0-dev.20220110.b4d497d","keywords":["oclif-plugin"],"author":"Microsoft","license":"MIT","_id":"@microsoft/bf-dispatcher@4.16.0-dev.20220110.b4d497d","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":"https://github.com/microsoft/botframework-cli/issues","dist":{"shasum":"05fcba9098bc85a5d7d99404f6d0f414973765ca","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.16.0-dev.20220110.b4d497d.tgz","fileCount":587,"integrity":"sha512-SNKerCfmNMYDZnvKFabcl+XJDF9EwdJBghK1F/BZe4IdXVmRE6zuTdsJXnGb5RTA35pFCqY7yIIaMtym0ruz+w==","signatures":[{"sig":"MEQCIBtYcZ2xn7Vh05vGLIVqFWkICE5ULFdxKjcqh5soY6O9AiA1SAJe3sku4mrdeIGnRCdP/ZlqmpvMj4Gh6eoNlyT00g==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJh2+ygCRA9TVsSAnZWagAAeboQAI1tDDp5N3gSCTxFboVr\nxDCb4nl0pmlVRHkj37Sp5YnqpQw5AuNzdtWZZPgrcFD93Jw8ldGdX6obxORk\nouuAyZL4mCLLkBbVhbTKbH6YcMSkLxDqHAnn/paUO6t5usFrUyKvXUHjRNwE\nwDiOPB7EuDwoLuXDwdmjUHeqv9+hOSSqiG0RIzTBjpXVPwok/Eaxennqqqe0\nOW5o/v2SoQgLPtgnRp7PAEkD2ct5HSXGI1G6C9WQrBxAF1BKnYbg6VFv/oQb\nc4GReTvxQ25TecfN7zB6hynDz0IAG69NbEdWngqaaEfY5cnbMrRkQrSknpyV\nM22MHg6zmdTImNqF3gH+aiteOykynf1z1s9Ve1/FEN1MuhWMTcRpkYau30IZ\np+66Kf1cjOty3jaw0MHLzJXBUoScVLPyBWoQnxbZDqOS8MwkVF/DgsOfI9nT\nHxZdrsntL70BG9gXR5AMBMR8jTtM7sdF7k9ySbL3EGpMLmWXtI2tCPdqjzqg\nuLsT66lQgGlFwAD2c3QhnZ+V8aU/G8aYVghBYZ8oPXJgpyBBlqBE+dCo46Qx\nxAJryE4OcXYwSthNrj5O3rayDB0xXHM3bsKXNucfbg5Rvgwhg6i15LkbvVE8\nOVA/o3cBZ+saPnD2pdz86+HyUbg1HhoQ5PDTIQb9RN0Drmj2fH3qNNmpDGkP\nwICx\r\n=nNtV\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20220110.b4d497d.tgz","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20220110.b4d497d.tgz","_integrity":"sha512-SNKerCfmNMYDZnvKFabcl+XJDF9EwdJBghK1F/BZe4IdXVmRE6zuTdsJXnGb5RTA35pFCqY7yIIaMtym0ruz+w==","repository":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","_npmVersion":"8.1.2","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"16.13.1","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.16.0-dev.20220110.b4d497d"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.16.0-dev.20220110.b4d497d_1641802912584_0.052250563775334324","host":"s3://npm-registry-packages"}},"4.16.0-dev.20220120.e933dfc":{"name":"@microsoft/bf-dispatcher","version":"4.16.0-dev.20220120.e933dfc","keywords":["oclif-plugin"],"author":"Microsoft","license":"MIT","_id":"@microsoft/bf-dispatcher@4.16.0-dev.20220120.e933dfc","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":"https://github.com/microsoft/botframework-cli/issues","dist":{"shasum":"1d7c0756a422e5638ef2a438a86fb3a2667bfb1e","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.16.0-dev.20220120.e933dfc.tgz","fileCount":587,"integrity":"sha512-yeeOTah1gllwO5y5EQUgp9ZQOk/prHZrHf+ccNwfBMVeTsAqPGPvqExE8KLh+kqqhjUcQmWUvgnmv2VfRhLupg==","signatures":[{"sig":"MEUCIQCSsReUVL9BHm+y/wY6jhMbiZU0wYwf5HZXKZgLlHbxqAIgeqTsEcvHq0mCU1E3MTe+YcscXZqWRtvC1JlFdC7iDf8=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJh6RwfCRA9TVsSAnZWagAAQ9oP+QDk9RbGw36Gbm36MNDq\nAr2pg5HuhZNRnG+NZzPRhKBdpiVd2SQrZlIBwfvaywTFJF2obt68GPpAiZQk\noGNcENvjY/XJxA7D6wLpWSR6nNBjKKOGWm7/hvqzPYMUNyQ6TEoK5m1W4fHA\njrsGpErDo6rIqtR7nYqlVYHHsLDaqmww9y03WDQSIUSrAojnV7uET2+A9P5V\nPHlm7SdIXwYqBQsE02wdE9s6XpVMHvEN/806xeosnOFe5KkOibZiI9ReFEQx\nM+chqxevT5gVDcq3INW80mT8v0SyQAaVgKP+AKzGIzcdDrJEiNfzGc27d8d6\nKhbhDITpCOA2Qt0S6zSIIEv2Sbe9D9Z5cfWFkTsVU/uP0F6k1eFCIE8VL+z9\nHT1L6DD80HVi5+PA5olglPXcCZ2imiE8UP07wa0u66eOnhIIIu4ImtVkVbet\nYN27BEPJnUdjl10PfcEyzTvBjw3lc5CRHqwiQlgnrhxIymCKV7IEuILVHhCM\ndSyNvPqfqFNOP/gkAm83cJ/NQTFbYKJvRMNJfpaevVkG3JHAUUHcK9dXDD7w\nPtGNd9uSLKscD4r2o920bTazvjyjH8sKhcFb0fNpPxbAoDFDzM1BBM26vk18\nP0zX29B6oII9Z2UXX7iGEflQ+iXgloALZA59X7BPp9Ni+Fntl18T24qNul0b\nvzDP\r\n=igGY\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20220120.e933dfc.tgz","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20220120.e933dfc.tgz","_integrity":"sha512-yeeOTah1gllwO5y5EQUgp9ZQOk/prHZrHf+ccNwfBMVeTsAqPGPvqExE8KLh+kqqhjUcQmWUvgnmv2VfRhLupg==","repository":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","_npmVersion":"8.1.2","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"16.13.2","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.16.0-dev.20220120.e933dfc"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.16.0-dev.20220120.e933dfc_1642667038891_0.8838557685518673","host":"s3://npm-registry-packages"}},"4.16.0-dev.20220228.5bb413a":{"name":"@microsoft/bf-dispatcher","version":"4.16.0-dev.20220228.5bb413a","keywords":["oclif-plugin"],"author":"Microsoft","license":"MIT","_id":"@microsoft/bf-dispatcher@4.16.0-dev.20220228.5bb413a","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":"https://github.com/microsoft/botframework-cli/issues","dist":{"shasum":"547ac3c901e2c0054303b407ac8025ff6bad988c","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.16.0-dev.20220228.5bb413a.tgz","fileCount":587,"integrity":"sha512-5k6eyBw2wz+BqGhOi7oe7CoLamT/JvisNl1noX8qLWIhaD+5cyfdn8fcICf52T40h3F9vDqEZPC+J7ZM4OKyBg==","signatures":[{"sig":"MEQCIGvp9/0QSD9A27/sT8P+oxSZ/ffyZcKZ4DpwvklMIz7FAiAqsgsRVrl3u0Qy55M8/50MKizLwLSIrGD2de1TAuVvHQ==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v4.10.10\r\nComment: https://openpgpjs.org\r\n\r\nwsFzBAEBCAAGBQJiHIXPACEJED1NWxICdlZqFiEECWMYAoorWMhJKdjhPU1b\r\nEgJ2Vmo6qxAAoQncMnHet53u4NHpz4zUOrmGRMjjFNxteWVsr1nUtr27a5P3\r\nQl4DOLUHQ7+gEnTvBsDLzIvqz/DKq7TrDFzW8OWRuVySMu1cxI+O470slp4S\r\nKi9CrXf9ogxLzQOQXE5yO4p6D4f/Kjht8MsU2bi6adYGScrMLPUkZrNSzW+6\r\nZ+mYoPXbGynopy+nNZ692PEw8mrFxDgbEtaOosCZklIEejqWdyF3nu4A7hMr\r\n71++U8wzru8nKKoAq1h+0I+VtMuRccifRLg2UyLRqogyoE7jdgRnvy7ajZZx\r\n0L8SFeWfJnl/CT88BmeHYrZghC8dPNYr205koSoDIiA0ZR9n3L9Aa3fPPGCF\r\nS7PbruhQGKY/8yQ+0U8fh4CetOO3Z1QB5MAIQ8rHKCnzv/wyqzy0TIL9yGEB\r\nQkJI1NTBjmQQuvNse+yI4cGG9QR8Y+8z5iHtjiMxgucgr8FfafUUH/PxRHqx\r\nQuXXF1sNGNLuY/sXoW95wc8wp0GO9rgzJSNuM7HeqbFg0sadbA9ApocJe9B8\r\nO+XatATWocELsRfIGqM1Y2LRaVj4WsAozsc8Cqy9ZeGSGrrYlEE2tfb6MLyW\r\nExl0K8TsnTaz1/Q0T/KNpnfE5iqdTx/rB+wD2HjVLQE0LIQCcOY14xprpVt1\r\nJwuoQMkyyx5lnhi2BhSD5/PzDU56ihsH0XE=\r\n=jTXz\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20220228.5bb413a.tgz","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20220228.5bb413a.tgz","_integrity":"sha512-5k6eyBw2wz+BqGhOi7oe7CoLamT/JvisNl1noX8qLWIhaD+5cyfdn8fcICf52T40h3F9vDqEZPC+J7ZM4OKyBg==","repository":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","_npmVersion":"8.3.1","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"16.14.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.16.0-dev.20220228.5bb413a"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.16.0-dev.20220228.5bb413a_1646036430948_0.1896585844764691","host":"s3://npm-registry-packages"}},"4.16.0-dev.20220525.147c370":{"name":"@microsoft/bf-dispatcher","version":"4.16.0-dev.20220525.147c370","keywords":["oclif-plugin"],"author":"Microsoft","license":"MIT","_id":"@microsoft/bf-dispatcher@4.16.0-dev.20220525.147c370","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":"https://github.com/microsoft/botframework-cli/issues","dist":{"shasum":"b082f88ebb73848c9a38ebfac8d0b851161a8133","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.16.0-dev.20220525.147c370.tgz","fileCount":587,"integrity":"sha512-+jEqVsJRk5M25ag8vW+Q44RHvdaiRcsrJNGpAixFhyzjUYhY+GvmxXcdRvUYhXpCROkjORhAfwxly5or08lC/g==","signatures":[{"sig":"MEUCICndmJKBf9ZwCegVjc5W1qiuJTxogm+2GGDuZukNYEh+AiEAgCEbxKbyrZIeJkWzAsjPhtFog4rb+DT9c1TUm9wwCJM=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v4.10.10\r\nComment: https://openpgpjs.org\r\n\r\nwsFzBAEBCAAGBQJijdktACEJED1NWxICdlZqFiEECWMYAoorWMhJKdjhPU1b\r\nEgJ2Vmp1KQ//aybArCzLe8230z5QYtRAPvfxa1VmzUr9BQJV88xh9V0i+nAB\r\nvuHuF4kY0sNBwowoQflbnZXDijpvsRC+ptHsA8bU3u1+mNRRjkiWk7Ik2eul\r\nC+7NhkHiy+UrakokpWWLiP62W8cNeH4Yhu/zLbeZK2DD2+qlLpUpR6+o/ZZJ\r\n+mTR1zX5azlIpRFTr5LNy6/5awN9FFxz0qE6Zcg2s+3+cPXKq/Vozprx2CdW\r\nGQ7j/Bpul9BgcYsRW5QYhLsVFQ4YYazW9jCYOcRZBfVTz5zyA01KZO/8WN4e\r\n9T/yHAs88fOuDmvfXpQgT8Jdvg/UqnPIMETgvTebx5fias/8DhSBtWycbaj0\r\nRPDcUU4IPGCIJjq8bu7HqODySwZs9go78dB8es+w1E19U3PTT7U52f+/nMfX\r\nGx6x7KIEE0q/wJl13gzXd/GWwN8IFxuJy2WhXysbHjYXVIrsyWkRN17ttbBc\r\nDEd/ATNuhV1uKkCr43v1XhB8pNFi/aKpa8f8BEsf8LerL0mTsQujhrWnhjwp\r\n1M1EBTjRj1VAAagFM7sxMg4Ij1PwyYmx+mqVWW7lbV+dg1Y4YdBZjz0TDXbf\r\n7neu6jh0L9MsW8KJAj2wYu04guc0LAq4A8NFXWlQWz3ZfMYfVXUDpqbNngTG\r\neYwNBvnQV4s68oSN11MftlkjHmjhEVv3yJM=\r\n=np7p\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20220525.147c370.tgz","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20220525.147c370.tgz","_integrity":"sha512-+jEqVsJRk5M25ag8vW+Q44RHvdaiRcsrJNGpAixFhyzjUYhY+GvmxXcdRvUYhXpCROkjORhAfwxly5or08lC/g==","repository":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","_npmVersion":"8.3.1","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"16.14.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.16.0-dev.20220525.147c370"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.16.0-dev.20220525.147c370_1653463341155_0.14591645908755346","host":"s3://npm-registry-packages"}},"4.16.0-dev.20220526.695cee5":{"name":"@microsoft/bf-dispatcher","version":"4.16.0-dev.20220526.695cee5","keywords":["oclif-plugin"],"author":"Microsoft","license":"MIT","_id":"@microsoft/bf-dispatcher@4.16.0-dev.20220526.695cee5","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":"https://github.com/microsoft/botframework-cli/issues","dist":{"shasum":"de07684a6700c43a068de716ff3a3bce3da4dc9d","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.16.0-dev.20220526.695cee5.tgz","fileCount":587,"integrity":"sha512-6nOv/Ncofqn7mLWSsoZsPWPxWO2JbZmQRMxxPD0VbOq+0SVBRQDHIChaJBq7AsEWUKHmcCfeJDwW9Z6eMjs2Fg==","signatures":[{"sig":"MEYCIQDDMcAp56K69LVWzseHIB4wiiCwpXa8koQ67bDkPvVvEAIhAOCeFnKKFLZo/Z1REcep2o689Pz6oYP7e89LTarn43oG","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v4.10.10\r\nComment: https://openpgpjs.org\r\n\r\nwsFzBAEBCAAGBQJijyovACEJED1NWxICdlZqFiEECWMYAoorWMhJKdjhPU1b\r\nEgJ2VmplpQ//VhNcMbJ4eOk1fWNJBVNZOV6pF8Nn96xJplho6Hbah9Z37IlK\r\ntMNidJSGPvi2ksgfUEEYJJMp9TN0AKQjfRYU3NGoOXisNKEmpUf8QcKtx61a\r\nXgk6TFNbVKUnpwl3Wj15i56hMpzBq8VhDOb96BVVXsjcxCVl4GWsTQtUNSmC\r\nh15vBBLtPMIbVSW910fAH/SBtvIsKfhCporxzr0Y/jfro8B+Gg4VaR8XmI1P\r\nSThm7CtXplalWfVSljtWfv+XlyDhhvW0OHPLV3ULYbqVNTUK3Fo6HQvccPAa\r\n7Jqu+No9TNk2dCJ8S+7qWDzMiAgBTNfDl7VYRBvKorbA7dAhnJ4K36uYGTbQ\r\nJusCN/f4f5U474vdn5/XbT5jSccL2ROVmhEbpVTxMhz7I0j+f0ytCN8OxYkz\r\nLD/Z0LrnzUpBiYKcVjr0/yJO/dIJg5qKqV8MlafEAp5Y09s9syeSU9rpWlhK\r\nDKQfVWYWK1J1RsuXgkHLoO6nrGXqS9dwGI53iJlWKA8pRyRgdtl+hPDJn3hp\r\ni4yuVmL3XFc/2LZI51hqdTyPuXYQUbH1cXVo8KLLHO4c7FFCS5UOm/TIcBX6\r\nZIaBSlkeKzsrtr5vUrDJk64W3n3aFhFqPrpRnqL3lMSWSnJqXbamKUzkS0wG\r\n5xTTZ5CrsPFt7PzIxJadu52Kh1TcwivwLFg=\r\n=U+86\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20220526.695cee5.tgz","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20220526.695cee5.tgz","_integrity":"sha512-6nOv/Ncofqn7mLWSsoZsPWPxWO2JbZmQRMxxPD0VbOq+0SVBRQDHIChaJBq7AsEWUKHmcCfeJDwW9Z6eMjs2Fg==","repository":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","_npmVersion":"8.3.1","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"16.14.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.16.0-dev.20220526.695cee5"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.16.0-dev.20220526.695cee5_1653549615532_0.057497753058912915","host":"s3://npm-registry-packages"}},"4.16.0-dev.20220725.a0ab7ca":{"name":"@microsoft/bf-dispatcher","version":"4.16.0-dev.20220725.a0ab7ca","keywords":["oclif-plugin"],"author":"Microsoft","license":"MIT","_id":"@microsoft/bf-dispatcher@4.16.0-dev.20220725.a0ab7ca","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":"https://github.com/microsoft/botframework-cli/issues","dist":{"shasum":"3a2ea5618e64c35ec5e2a381a86d4d271ba2218f","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.16.0-dev.20220725.a0ab7ca.tgz","fileCount":587,"integrity":"sha512-HsclFkCXz+xrTGhnFHkjF5VRpWZscHuSdx3lZVk3BSfMYDHPeiv0J8XD6UheKYoeRGQ8YvkZdvEXNsnaGnh5WQ==","signatures":[{"sig":"MEQCIEvR/HVqtQoKHf6nKCs6U2/pmtfmOH8imm7nlV6pjo+kAiB271pnNXI7JNCJDLVfy5IuiBKFEHMJA3WHlnMqAUV+kw==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v4.10.10\r\nComment: https://openpgpjs.org\r\n\r\nwsFzBAEBCAAGBQJi3kSTACEJED1NWxICdlZqFiEECWMYAoorWMhJKdjhPU1b\r\nEgJ2Vmo47xAAniI5+jz/RHc21LlkrrLSs7w+nc913kngGdJBLs6TVyXXfgsK\r\nPqvZtAVyER7KpY7BTzXBSmuyb6wOyIePfxCjwszB62jSoaOY885ZdT6rOemc\r\n+gwT3koR8dS/iETG7p7MxmPpE39Bzplrex3AsLXyx734cAGjpnFAVnuR1aZd\r\n9C2j/xVIemrrp3pzXrAR6pLyuQ8BGqfaRwqEa5mBaW5kF7BZeaW83qhSoONM\r\ntcBBPCPpf8+KRhAf5F5lFESqag3wjxbWtrZzGShIKjARGE5TBy2Q5caOTpRG\r\nTaSFg9zvbrfQe1x8HsHcCFSlwT/jZH9ZQ5iAzuvqQTQ12uB3eS2GDCMnvUXg\r\nYrVI6G/222/v0ghNlISto+Bl2wMbINUg9nYhO1UVncnIXDKcpiwXWDkEoAR3\r\nPSQytiAKEWYdv0mm596rOdXe71DwRBPXCWL0w9TnnNS45+k4B/4biQmPOH1G\r\nAicv/Bxye7aVZsf8CClmP14j72yYquHDClizNZeBSmn6xSsmO3cTsYdX3mLx\r\nmDY/5MaDqK+0qorpEYR2EsMrM1y+LRwgfGsNhfMlJUISx54oSJ1+arVoXBV7\r\nt2duUobr8wVTZJBF7hxgK4SeqVq0RM3HuEA70dnt23Kowk1CnxE0g4PD1RIx\r\nbRoflBHhRRjQsrh6sKEwWSrodT6C8qZDZ0Q=\r\n=+RAI\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20220725.a0ab7ca.tgz","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20220725.a0ab7ca.tgz","_integrity":"sha512-HsclFkCXz+xrTGhnFHkjF5VRpWZscHuSdx3lZVk3BSfMYDHPeiv0J8XD6UheKYoeRGQ8YvkZdvEXNsnaGnh5WQ==","repository":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","_npmVersion":"8.3.1","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"16.14.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.16.0-dev.20220725.a0ab7ca"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.16.0-dev.20220725.a0ab7ca_1658733714945_0.242170788975836","host":"s3://npm-registry-packages"}},"4.16.0-dev.20221107.64f07e4":{"name":"@microsoft/bf-dispatcher","version":"4.16.0-dev.20221107.64f07e4","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.16.0-dev.20221107.64f07e4","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"8991aa30d22b4b6a926ee2c85d50f2e5fce5b1ed","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.16.0-dev.20221107.64f07e4.tgz","fileCount":587,"integrity":"sha512-M/JaA75hGIqab5aPo6660CbTod/UC5n5eiJlDA9wdeZyQVCCIFiYHTh4kTkAteh+q+NhhPFawNVocrYiEG6D2g==","signatures":[{"sig":"MEUCICZluw//10sb4O+u07BXAsyXE6xIGK10LOi+vNoDZq3TAiEApQkqK/dJNCYvpcEIYIr37iT1PGjj4MR65WtgB80jbLg=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v4.10.10\r\nComment: https://openpgpjs.org\r\n\r\nwsFzBAEBCAAGBQJjaMBwACEJED1NWxICdlZqFiEECWMYAoorWMhJKdjhPU1b\r\nEgJ2Vmq0xA//bP8Qs3FSSTWbsfbbh30jwze35TJkwDbmi7tRS7wtpqIuxUXj\r\n+VGsLHjufZVdWUYYZEATbWLqdN3qnDWbqmmuEN+R7+05KPH3oBBbDOykAXli\r\nJ5UJ71acNArQ4/3koetkDSACSyTu5czhp4h5RyeJGH9cR6/mgeBnC4zmikW7\r\nUCRID9WdA0aJWK3CLKaku62/5nSjP0VagaRqKs+IoHQSnwqBphIPMymZIK9D\r\nAI8xXdA3yq69HgclQIRhHpPegbIe8lGDctQtG28b3iniq6WaKyxj8UkdWonl\r\n+x/EGlPALxIsy9cKWsfupKfykfFz8tQyO/vj2RmSaulw+EYKeu7cHA0k+A14\r\ngjuO5JHlOTF1V1tVzLv9egE9DbTnip1IAeYQPVM2iE3PtNgejPBeZ/bCUWKV\r\nBsroZ+H3HqfH+QSkCnduqvh7TCtc+SjTvjuUkHa7Eb3jy1NoXYdX/33OSZ+i\r\nRjwUbZJXhSvDRnAVh5crCxG9ZXWQR+yEzySedN7O1RddORstpZnrxrQCoRQJ\r\ngz8fJel4ExZ1ecGMQtU89Nbleji5ho3SENInCmtynJxdJY2o6vz7gt5vu+6C\r\nMaX5eTNzYyGbvrq+Mtjp+xiCuL0N+S6h3kmeZcl6aejW/GegrYQWROMMN3x5\r\nOkLPFbRYDW1orJ2K/iUEx1JYGH71aPGI0IA=\r\n=5uVK\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20221107.64f07e4.tgz","types":"./lib/index.d.ts","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20221107.64f07e4.tgz","_integrity":"sha512-M/JaA75hGIqab5aPo6660CbTod/UC5n5eiJlDA9wdeZyQVCCIFiYHTh4kTkAteh+q+NhhPFawNVocrYiEG6D2g==","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git#master","type":"git"},"_npmVersion":"8.19.2","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"16.18.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.16.0-dev.20221107.64f07e4"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.16.0-dev.20221107.64f07e4_1667809391957_0.26665441220864583","host":"s3://npm-registry-packages"}},"4.16.0-dev.20221111.64f07e4":{"name":"@microsoft/bf-dispatcher","version":"4.16.0-dev.20221111.64f07e4","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.16.0-dev.20221111.64f07e4","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"c006772312bf6975126b5d3042b037cd8d4e7a6f","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.16.0-dev.20221111.64f07e4.tgz","fileCount":587,"integrity":"sha512-DA3kGS8/otYmnH6q//dueuHVzvfhL5mksi1qBrJvuD2Z3B2wmQK738K9SJy3a40G8KRJVYoRMkADlVSk3sV0Fg==","signatures":[{"sig":"MEUCIQD/SlF/x66Nj67g9pxBrjgnojIc9/RB6iDW+yarMSprJgIgCZfr61lUNP467m4sF4rRVShli5U0R6QVH4t8+TnPGlM=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v4.10.10\r\nComment: https://openpgpjs.org\r\n\r\nwsFzBAEBCAAGBQJjbta2ACEJED1NWxICdlZqFiEECWMYAoorWMhJKdjhPU1b\r\nEgJ2VmrSZQ/8DxTQVROWloXq6B3YAiybVGgo8WbQGlTS/g3dqvVl2F77dH1L\r\nPAYss5cLNYQpSdxg3bx44PrYZBNPqYOzbHcuSoCkTN+dAKT063O4HMd/sTnD\r\nKOZe6SuKtrmdc/7FKH+866DJwgyD4f3WOYWqTyKdwnCsJiLlSUSmTvH3Y8LJ\r\njoxdZH7Qr9JFVZJ9Yqi4b6zz8qBWyeqQCoAAG17ABjz0ZcN3zkaJ38J/wfjv\r\nCpaUB37nX6OSdIV+Fb28WUg7qazSyuSt6vN/jmuEumyqzIECM4AfoMsSQ4Tm\r\nmBJrmQAWjfCuorGCBZPtAlV0tttuFniITflRpgjKHGXjVVpE4fHvfuAWiENt\r\nltCl0vbAafhtc4APwMVEbCFPgG0iinKCsn3PrV9+RjiBU/H/vDag6yI0ijDn\r\nRyuemgyBPsHj9rtrmuYYJoaqBea3X+SqBcpD/TGBMglNt33+Y97KMqB3k/bm\r\nH6QBYiKdw/rodUwo7iR/kTJHZzZrwTY1ac00Mki3AHzrZSB84XlzQ41lDsvF\r\n15I7mzAmoApzKRDRlbW3XBp4awcQlNr9IVNYUwc4UxYa4AG9hcIsER1k/+ab\r\n0r4FNP210ECIVCC9NjsREIS5ljTvSsxNPq0oYcuGmRzgCaH/r3FQzH1V1PFS\r\n8WoYCzFr9Q1jIkSP7IjsrB+4HBECG8rvoGE=\r\n=DtJd\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20221111.64f07e4.tgz","types":"./lib/index.d.ts","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20221111.64f07e4.tgz","_integrity":"sha512-DA3kGS8/otYmnH6q//dueuHVzvfhL5mksi1qBrJvuD2Z3B2wmQK738K9SJy3a40G8KRJVYoRMkADlVSk3sV0Fg==","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git#master","type":"git"},"_npmVersion":"8.19.2","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"16.18.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.16.0-dev.20221111.64f07e4"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.16.0-dev.20221111.64f07e4_1668208309926_0.7674699265765996","host":"s3://npm-registry-packages"}},"4.16.0-dev.20221114.64f07e4":{"name":"@microsoft/bf-dispatcher","version":"4.16.0-dev.20221114.64f07e4","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.16.0-dev.20221114.64f07e4","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"f28a36f38d93dd359a5f263529f855082f1403c8","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.16.0-dev.20221114.64f07e4.tgz","fileCount":587,"integrity":"sha512-/4MLKHwq2rtRQKI8WJeGhyoH9zwSU8h8E2JUNJZwtycK6ZCSliAoJr+Hq3Gmec+nA0LUaT09cEJ/DQQXdPJUPQ==","signatures":[{"sig":"MEQCIHIxquq78ihUyWjFDMMgck0NMvXza6UDr0VdsHDIEDgsAiBTv92iDwKoqSDwrotOpRzILqPcsETf1JEcQgir+goVuQ==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v4.10.10\r\nComment: https://openpgpjs.org\r\n\r\nwsFzBAEBCAAGBQJjcfr/ACEJED1NWxICdlZqFiEECWMYAoorWMhJKdjhPU1b\r\nEgJ2Vmpr4w/9EwJuvvYXfLCsPTk6nrU/TBor/Sks7Denc7/RmrNOrZIB4A6I\r\nJtfiVa1ssN0pKGjs7wz2wOhdsi2p2GYBKc1VbwDGQ1ccfpX5+gqJ4W+qZPgv\r\nSTmwmb/F0JTiwV/ay9MtC3TMgZtfLYi+mnrqAey3vtAKPODBxeNdU5TwCxd/\r\nM5tC/RdYy3LbsjV1bbfU9yGjOuCV2IYZm9JmJCVdeUPnhmEZO8Jk0nMJZ1Ie\r\nzhgS7s+9zhjCIcLlXsRx69pvIHvqASuXcPV+Zh7hypUONHcl7HG5jQ+faWRE\r\nxXldptWxE+NuU178dX8WiDuhha3fsBYoEgCofN8HZY71gumnELU/lf98jX4v\r\n+MH6UOaqFkInwK8NmmCcfM09Hwj+a1UdTQObm2OP9IsaeEvNaexxZHs2EHcM\r\nizy+egzfOPYxuSXyS4IKS7zdi6RGr1YzpeNl9DO9im2LaFbxY5x122Id/9+A\r\n1yR1bDrA7T/gzfz3CDXGNLbxp/BEtx3indZk7eKG/gn6e9H7c9N9bBL+11gg\r\n3SD8rrFNw1XWeH/bkQEWnsU/os+0QA5Lpi6YMxHkjD9TbmrVM2cwDM2h7Md2\r\nxdtY7S4TcERsvoRFjX1CiixRsqu+hkJJDuGuI3NeRO7WcHAF9Ag8GrBfKn56\r\nvoGXI25XYBjz9YhDxAfERJo+uHPi5XX0R9I=\r\n=1LO8\r\n-----END PGP SIGNATURE-----\r\n"},"main":"lib/index.js","_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20221114.64f07e4.tgz","types":"./lib/index.d.ts","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20221114.64f07e4.tgz","_integrity":"sha512-/4MLKHwq2rtRQKI8WJeGhyoH9zwSU8h8E2JUNJZwtycK6ZCSliAoJr+Hq3Gmec+nA0LUaT09cEJ/DQQXdPJUPQ==","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git#master","type":"git"},"_npmVersion":"8.19.2","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"16.18.0","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.16.0-dev.20221114.64f07e4"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.16.0-dev.20221114.64f07e4_1668414206692_0.6562020601301846","host":"s3://npm-registry-packages"}},"4.16.0-dev.20231122.4561d55":{"name":"@microsoft/bf-dispatcher","version":"4.16.0-dev.20231122.4561d55","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-dispatcher@4.16.0-dev.20231122.4561d55","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"cmullins","email":"cmullins@gmail.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"88719b03ab846de3a4b22ed523bbc3df15fc7fc4","tarball":"https://registry.npmjs.org/@microsoft/bf-dispatcher/-/bf-dispatcher-4.16.0-dev.20231122.4561d55.tgz","fileCount":587,"integrity":"sha512-YT50Q6rXTLRBltVCvHgsw/dB4HPLK9zXsxFVEjOCPW+K7zHwm/AFU6xQdgk94ecry8Cv3b9rTnX3XMPVn5WW5w==","signatures":[{"sig":"MEUCIAimtfF0+F0140q+YV95wDTkSmT6+JdRIUGudVuCwSYyAiEA7r06MM9cvdcaqkMOdIX/XiGVEWB6pvqmu926rD9uxEA=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":2198673},"main":"lib/index.js","_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20231122.4561d55.tgz","types":"./lib/index.d.ts","readme":"This package is intended for Microsoft use only. It is not designed to be consumed as an independent package.\r\n\r\n@microsoft/bf-dispatcher\r\n========================\r\n\r\n'bf-dispatcher' is a generic NLP intent classification package/library.\r\nIt currently can do:\r\n\r\n> Sampling\r\n    > Bootstrap resampling\r\n    > Auto active learning down sampling\r\n    > Stratified down sampling\r\n\r\n> Evaluation and Reporting\r\n    > Cross Validation against a training set\r\n    > Test again a test set\r\n\r\nTo support these features, it internally contains a Softmax Regression (MaxEnt) learner that can consume sparse text features and train models to support auto-active-learning down-sampling and cross validation. It can also produce model quality reports.\r\n\r\nCurrently, 'bf-dispatcher' can consume two forms of input sources: LU and TSV columnar files. It uses the bf-lu package (https://github.com/microsoft/botframework-cli/tree/master/packages/lu) to parse and load a .lu file as input.\r\n\r\nTo demonstrate the auto active learning process, there are some example functions implemented in\r\n\"src/model/supervised/classifier/auto_active_learning/AppAutoActiveLearner.ts\" that can do the following:\r\n    a) consume a LU or columnar TSV file,\r\n    b) use a bootstrap resampler to select training instances based on a prior label/instance (intent/utterance)\r\n       distribition,\r\n    c) iterate through batches of input utterance/label records, and train models to select most relevant utterance/intent\r\n       pairs through an auto active learning process, and\r\n    d) use a stratifier sampler to limit the training size.\r\n\r\nBelow are some examples of using the AutoActiveLearner class.\r\n```\r\n    /**\r\n     * This function can read a LU file with intent and utterance data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param luContent - a .lu file content in string form as input.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithLuContent(\r\n        luContent: string,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newLuData\": LuData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let luData: LuData =\r\n            await LuData.createLuData(\r\n                luContent,\r\n                new NgramSubwordFeaturizer(),\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    luData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`luData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(luData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const luDataBootstrapSampled: Data =\r\n                await luData.createDataFromSamplingExistingDataUtterances(\r\n                    luData,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- labelColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- textColumnIndex,\r\n                    -1, // ---- NOTE-NO-NEED-FOR-LuData ---- linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            luData = luDataBootstrapSampled as LuData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            luData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = luData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            luData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            luData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                luData.getFeaturizerLabels(),\r\n                luData.getFeaturizerLabelMap(),\r\n                luData.getFeaturizer().getNumberLabels(),\r\n                luData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newLuData: LuData = await LuData.createLuDataFromFilteringExistingLuDataUtterances(\r\n            luData,\r\n            new Set<number>(aalSampledInstanceIndexArray),\r\n            false);\r\n        return {\r\n            newLuData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n\r\n    /**\r\n     * This function can read a TSV columnar file with label and text data and run through 3 steps of\r\n     * sampling processes:\r\n     * 0) Bootstrap Resampling\r\n     * 1) Auto Active Learning Sampling\r\n     * 2) Stratified Sampling\r\n     *\r\n     * @param columnarContent - content of a TSV columnar file in string form as input.\r\n     * @param labelColumnIndex - label/intent column index.\r\n     * @param textColumnIndex - text/utterace column index.\r\n     * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n     * @param doBootstrapResampling - boolean flag to activate bootstrap resampling (BRS) logic or not.\r\n     * @param brsDistribution - explicit distribution to control bootstrap resampling process\r\n     * @param doAutoActiveLearning - boolean flag to activate auto active leaning (AAL) process or not.\r\n     * @param aalLimitInitialNumberOfInstancesPerCategory - AAL initial number of instances per category/label/intent.\r\n     * @param aalNumberOfInstancesPerIteration - AAL number of instances processed per iterations.\r\n     * @param aalInstanceSelectionThreshold - AAL threshold to pick a tested instance for training in next iteration\r\n     * @param learnerParameterEpochs - AAL Softmax Regression learner parameter - number of epochs.\r\n     * @param learnerParameterMiniBatchSize - AAL Softmax Regression learner parameter - mini-batch size.\r\n     * @param learnerParameterL1Regularization - AAL Softmax Regression learner parameter - L1 regularization.\r\n     * @param learnerParameterL2Regularization - AAL Softmax Regression learner parameter - L2 regularization.\r\n     * @param learnerParameterLossEarlyStopRatio - AAL Softmax Regression learner parameter - early stop ratio.\r\n     * @param learnerParameterLearningRate - AAL Softmax Regression learner parameter - learning rate.\r\n     * @param learnerParameterToCalculateOverallLossAfterEpoch - AAL Softmax Regression learner parameter - flag\r\n     * @param limitingSampleSize - sample size controled by a final stratified sampling process.\r\n     */\r\n    public static async mainAutoActiveLearnerWithColumnarContent(\r\n        columnarContent: string,\r\n        labelColumnIndex: number,\r\n        textColumnIndex: number,\r\n        linesToSkip: number,\r\n        doBootstrapResampling: boolean =\r\n            AppAutoActiveLearner.defaultDoBootstrapResampling,\r\n        brsDistribution: TMapStringKeyGenericValue<number> =\r\n            DictionaryMapUtility.newTMapStringKeyGenericValue<number>(),\r\n        doAutoActiveLearning: boolean =\r\n            AutoActiveLearner.defaultDoAutoActiveLearning,\r\n        aalLimitInitialNumberOfInstancesPerCategory: number =\r\n            AutoActiveLearner.defaultAalLimitInitialNumberOfInstancesPerCategory,\r\n        aalNumberOfInstancesPerIteration: number =\r\n            AutoActiveLearner.defaultAalNumberOfInstancesPerIteration,\r\n        aalInstanceSelectionThreshold: number =\r\n            AutoActiveLearner.defaultAalInstanceSelectionThreshold,\r\n        learnerParameterEpochs: number =\r\n            AppSoftmaxRegressionSparse.defaultEpochs,\r\n        learnerParameterMiniBatchSize: number =\r\n            AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n        learnerParameterL1Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n        learnerParameterL2Regularization: number =\r\n            AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n        learnerParameterLossEarlyStopRatio: number =\r\n            AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n        learnerParameterLearningRate: number =\r\n            AppSoftmaxRegressionSparse.defaultLearningRate,\r\n        learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n            true,\r\n        limitingSampleSize: number =\r\n            DefaultLimitingSampleSize): Promise<{\r\n            \"newColumnarData\": ColumnarData,\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"seedingInstanceIndexArrayInitial\": number[],\r\n            }> {\r\n        // -----------------------------------------------------------------------\r\n        let columnarData: ColumnarData =\r\n            ColumnarData.createColumnarData(\r\n                columnarContent,\r\n                new NgramSubwordFeaturizer(),\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                true);\r\n        // -----------------------------------------------------------------------\r\n        if (doBootstrapResampling) {\r\n            const bootstrapSamplerKeyMap: BootstrapSamplerKeyMapDistribution<number> =\r\n                new BootstrapSamplerKeyMapDistribution<number>(\r\n                    brsDistribution,\r\n                    columnarData.getIntentInstanceIndexMapArray());\r\n            // ---- NOTE-FOR-REFERENCE ---- const bootstrapSamplerKeyMap: BootstrapSamplerKeyMap<number> =\r\n            // ---- NOTE-FOR-REFERENCE ----     new BootstrapSamplerKeyMap(data.getIntentInstanceIndexMapArray());\r\n            Utility.debuggingLog(`columnarData.getIntentInstanceIndexMapArray()=` +\r\n                `${Utility.mapToJsonSerialization(columnarData.getIntentInstanceIndexMapArray())}`);\r\n            Utility.debuggingLog(`bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()=` +\r\n                `${bootstrapSamplerKeyMap.computeSamplingNumberInstancesPerLabel()}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- const samplingIndexArrayGenerator =\r\n            // ---- NOTE-FOR-DEBUGGING ----     bootstrapSamplerKeyMap.sampleInstances();\r\n            // ---- NOTE-FOR-DEBUGGING ---- for (const element of samplingIndexArrayGenerator) {\r\n            // ---- NOTE-FOR-DEBUGGING ----     Utility.debuggingLog(`element of samplingIndexArrayGenerator=` +\r\n            // ---- NOTE-FOR-DEBUGGING ----         `${element}`);\r\n            // ---- NOTE-FOR-DEBUGGING ---- }\r\n            const samplingIndexArray: number[] =\r\n                [...bootstrapSamplerKeyMap.sampleInstances()];\r\n            Utility.debuggingLog(`samplingIndexArray.length=` +\r\n                `${samplingIndexArray.length}`);\r\n            const columnarDataBootstrapSampled: Data =\r\n                await columnarData.createDataFromSamplingExistingDataUtterances(\r\n                    columnarData,\r\n                    labelColumnIndex,\r\n                    textColumnIndex,\r\n                    linesToSkip,\r\n                    samplingIndexArray,\r\n                    false);\r\n            columnarData = columnarDataBootstrapSampled as ColumnarData;\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const results =\r\n            columnarData.collectSmallUtteranceIndexSetCoveringAllIntentEntityLabels();\r\n        const smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            results.smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels;\r\n        const smallUtteranceIndexSetCoveringAllIntentEntityLabels: Set<number> =\r\n            results.smallUtteranceIndexSetCoveringAllIntentEntityLabels;\r\n        const remainingUtteranceIndexSet: Set<number> =\r\n            results.remainingUtteranceIndexSet;\r\n        Utility.debuggingLog(`smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(smallUtteranceIndexEntityTypeMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels=` +\r\n            `${Utility.setToJsonSerialization(smallUtteranceIndexSetCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet=` +\r\n            `${Utility.setToJsonSerialization(remainingUtteranceIndexSet)}`);\r\n        Utility.debuggingLog(`smallUtteranceIndexSetCoveringAllIntentEntityLabels.size=` +\r\n            `${smallUtteranceIndexSetCoveringAllIntentEntityLabels.size}`);\r\n        Utility.debuggingLog(`remainingUtteranceIndexSet.size=` +\r\n            `${remainingUtteranceIndexSet.size}`);\r\n        // -------------------------------------------------------------------\r\n        if (!doAutoActiveLearning) {\r\n            aalLimitInitialNumberOfInstancesPerCategory = -1;\r\n        }\r\n        const resultsInitialSampling: {\r\n            \"seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels\": Map<string, Set<number>>,\r\n            \"candidateUtteranceIndexSetSampled\": Set<number>,\r\n            \"candidateUtteranceIndexSetRemaining\": Set<number>,\r\n            } = columnarData.collectUtteranceIndexSetSeedingIntentTrainingSet(\r\n                smallUtteranceIndexIntentMapCoveringAllIntentEntityLabels,\r\n                remainingUtteranceIndexSet,\r\n                aalLimitInitialNumberOfInstancesPerCategory);\r\n        const seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: Map<string, Set<number>> =\r\n            resultsInitialSampling.seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels;\r\n        const candidateUtteranceIndexSetSampled: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetSampled;\r\n        const candidateUtteranceIndexSetRemaining: Set<number> =\r\n            resultsInitialSampling.candidateUtteranceIndexSetRemaining;\r\n        Utility.debuggingLog(`seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${Utility.stringMapSetToJson(seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetSampled)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining=` +\r\n            `${Utility.setToJsonSerialization(candidateUtteranceIndexSetRemaining)}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetSampled.size=` +\r\n            `${candidateUtteranceIndexSetSampled.size}`);\r\n        Utility.debuggingLog(`candidateUtteranceIndexSetRemaining.size=` +\r\n            `${candidateUtteranceIndexSetRemaining.size}`);\r\n        const countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels: number =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number, entry: [string, Set<number>]) =>\r\n                accumulation + entry[1].size, 0);\r\n        Utility.debuggingLog(`countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels=` +\r\n            `${countSeedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingUtteranceIndexArray: number[] =\r\n            [...seedingUtteranceIndexIntentMapCoveringAllIntentEntityLabels].reduce(\r\n                (accumulation: number[], entry: [string, Set<number>]) =>\r\n                accumulation.concat(Array.from(entry[1])), []);\r\n        Utility.debuggingLog(`seedingUtteranceIndexArray.length=` +\r\n            `${seedingUtteranceIndexArray.length}`);\r\n        // -------------------------------------------------------------------\r\n        const seedingInstanceIndexArray: number[] =\r\n            Utility.cloneArray(seedingUtteranceIndexArray);\r\n        const intentLabelIndexArray: number[] =\r\n            columnarData.getIntentLabelIndexArray();\r\n        const utteranceFeatureIndexArrays: number[][] =\r\n            columnarData.getUtteranceFeatureIndexArrays();\r\n        const autoActiveLearner: AutoActiveLearner =\r\n            new AutoActiveLearner(\r\n                doAutoActiveLearning,\r\n                aalLimitInitialNumberOfInstancesPerCategory,\r\n                aalNumberOfInstancesPerIteration,\r\n                aalInstanceSelectionThreshold,\r\n                learnerParameterEpochs,\r\n                learnerParameterMiniBatchSize,\r\n                learnerParameterL1Regularization,\r\n                learnerParameterL2Regularization,\r\n                learnerParameterLossEarlyStopRatio,\r\n                learnerParameterLearningRate,\r\n                learnerParameterToCalculateOverallLossAfterEpoch);\r\n        const learned: {\r\n            \"seedingInstanceIndexArray\": number[],\r\n            \"learner\": SoftmaxRegressionSparse,\r\n            } = autoActiveLearner.learn(\r\n                columnarData.getFeaturizerLabels(),\r\n                columnarData.getFeaturizerLabelMap(),\r\n                columnarData.getFeaturizer().getNumberLabels(),\r\n                columnarData.getFeaturizer().getNumberFeatures(),\r\n                intentLabelIndexArray,\r\n                utteranceFeatureIndexArrays,\r\n                seedingInstanceIndexArray,\r\n                Array.from(candidateUtteranceIndexSetRemaining));\r\n        let aalSampledInstanceIndexArray: number[] =\r\n            learned.seedingInstanceIndexArray;\r\n        const learner: SoftmaxRegressionSparse =\r\n            learned.learner;\r\n        // -----------------------------------------------------------------------\r\n        const numberInstancesPreSelected: number =\r\n            seedingUtteranceIndexArray.length;\r\n        if (limitingSampleSize > numberInstancesPreSelected) {\r\n            limitingSampleSize -= numberInstancesPreSelected;\r\n            const reservoirArraySampler: ReservoirArraySampler<number> = new ReservoirArraySampler(\r\n                aalSampledInstanceIndexArray,\r\n                numberInstancesPreSelected);\r\n            aalSampledInstanceIndexArray =\r\n                [...reservoirArraySampler.sampleInstances(limitingSampleSize)];\r\n        }\r\n        // -----------------------------------------------------------------------\r\n        const newColumnarData: ColumnarData =\r\n            ColumnarData.createColumnarDataFromFilteringExistingColumnarDataUtterances(\r\n                columnarData,\r\n                labelColumnIndex,\r\n                textColumnIndex,\r\n                linesToSkip,\r\n                new Set<number>(aalSampledInstanceIndexArray),\r\n                false);\r\n        return {\r\n            newColumnarData,\r\n            learner,\r\n            seedingInstanceIndexArray: aalSampledInstanceIndexArray,\r\n            seedingInstanceIndexArrayInitial: seedingUtteranceIndexArray };\r\n        // -----------------------------------------------------------------------\r\n    }\r\n```\r\n\r\nIn \"src/model/evaluation/cross_validation/AppCrossValidator.ts\", there are some example functions that\r\ndemonstrates how to use 'bf-dispatcher' to run cross validation and evaluate model performance.\r\n\r\n```\r\n/**\r\n * This function consumes a LU file content as input and run cross validation (CV) to evaluate models trained from\r\n * the input label/text (intent/utterance) instance set.\r\n *\r\n * @param luContent - input LU file content as input.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport async function mainCrossValidatorWithLuContent(\r\n    luContent: string,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): Promise<CrossValidator> {\r\n    // -----------------------------------------------------------------------\r\n    const luData: LuData =\r\n        await LuData.createLuData(\r\n            luContent,\r\n            new NgramSubwordFeaturizer(),\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        luData.getIntents();\r\n    const utterances: string[] =\r\n        luData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        luData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        luData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            luData.getFeaturizerLabels(),\r\n            luData.getFeaturizerLabelMap(),\r\n            luData.getFeaturizer().getNumberLabels(),\r\n            luData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            luData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n\r\n/**\r\n * This function consumes a columnar TSV file content as input and run cross validation (CV) to\r\n * evaluate models trained from the input label/text (intent/utterance) instance set.\r\n *\r\n * @param columnarContent - content of a TSV columnar file in string form as input.\r\n * @param labelColumnIndex - label/intent column index.\r\n * @param textColumnIndex - text/utterace column index.\r\n * @param linesToSkip - number of header lines skipped before processing each line as an instance record.\r\n * @param numberOfCrossValidationFolds - number of cross validation (CV) folds.\r\n * @param learnerParameterEpochs - CV Softmax Regression Learner parameter - number of epochs\r\n * @param learnerParameterMiniBatchSize - CV Softmax Regression learner parameter - mini-batch size.\r\n * @param learnerParameterL1Regularization - CV Softmax Regression learner parameter - L1 regularization.\r\n * @param learnerParameterL2Regularization - CV Softmax Regression learner parameter - L2 regularization.\r\n * @param learnerParameterLossEarlyStopRatio - CV Softmax Regression learner parameter - early stop ratio.\r\n * @param learnerParameterLearningRate - CV Softmax Regression learner parameter - learning rate.\r\n * @param learnerParameterToCalculateOverallLossAfterEpoch - CV Softmax Regression learner parameter - flag\r\n */\r\nexport function mainCrossValidatorWithColumnarContent(\r\n    columnarContent: string,\r\n    labelColumnIndex: number,\r\n    textColumnIndex: number,\r\n    linesToSkip: number,\r\n    numberOfCrossValidationFolds: number =\r\n        CrossValidator.defaultNumberOfCrossValidationFolds,\r\n    learnerParameterEpochs: number =\r\n        AppSoftmaxRegressionSparse.defaultEpochs,\r\n    learnerParameterMiniBatchSize: number =\r\n        AppSoftmaxRegressionSparse.defaultMiniBatchSize,\r\n    learnerParameterL1Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL1Regularization,\r\n    learnerParameterL2Regularization: number =\r\n        AppSoftmaxRegressionSparse.defaultL2Regularization,\r\n    learnerParameterLossEarlyStopRatio: number =\r\n        AppSoftmaxRegressionSparse.defaultLossEarlyStopRatio,\r\n    learnerParameterLearningRate: number =\r\n        AppSoftmaxRegressionSparse.defaultLearningRate,\r\n    learnerParameterToCalculateOverallLossAfterEpoch: boolean =\r\n        true): CrossValidator {\r\n    // -----------------------------------------------------------------------\r\n    const columnarData: ColumnarData =\r\n        ColumnarData.createColumnarData(\r\n            columnarContent,\r\n            new NgramSubwordFeaturizer(),\r\n            labelColumnIndex,\r\n            textColumnIndex,\r\n            linesToSkip,\r\n            true);\r\n    // -----------------------------------------------------------------------\r\n    if (!numberOfCrossValidationFolds) {\r\n        numberOfCrossValidationFolds = CrossValidator.defaultNumberOfCrossValidationFolds;\r\n    }\r\n    // -------------------------------------------------------------------\r\n    const intents: string[] =\r\n        columnarData.getIntents();\r\n    const utterances: string[] =\r\n        columnarData.getUtterances();\r\n    const intentLabelIndexArray: number[] =\r\n        columnarData.getIntentLabelIndexArray();\r\n    const utteranceFeatureIndexArrays: number[][] =\r\n        columnarData.getUtteranceFeatureIndexArrays();\r\n    assert(intentLabelIndexArray, \"intentLabelIndexArray is undefined.\");\r\n    assert(utteranceFeatureIndexArrays, \"utteranceFeatureIndexArrays is undefined.\");\r\n    const crossValidator: CrossValidator =\r\n        new CrossValidator(\r\n            columnarData.getFeaturizerLabels(),\r\n            columnarData.getFeaturizerLabelMap(),\r\n            columnarData.getFeaturizer().getNumberLabels(),\r\n            columnarData.getFeaturizer().getNumberFeatures(),\r\n            intents,\r\n            utterances,\r\n            intentLabelIndexArray,\r\n            utteranceFeatureIndexArrays,\r\n            columnarData.getIntentInstanceIndexMapArray(),\r\n            numberOfCrossValidationFolds,\r\n            learnerParameterEpochs,\r\n            learnerParameterMiniBatchSize,\r\n            learnerParameterL1Regularization,\r\n            learnerParameterL2Regularization,\r\n            learnerParameterLossEarlyStopRatio,\r\n            learnerParameterLearningRate,\r\n            learnerParameterToCalculateOverallLossAfterEpoch);\r\n    return crossValidator;\r\n    // -----------------------------------------------------------------------\r\n}\r\n```\r\n\r\nAt last, 'bf-dispatcher' also contains several model performance reporter classes in \"src/model/evaluation/report\" For now, there are three report classes:\r\n    a) DataProfileReporter.ts: consume a data file and report label distribution and feature distribution. Some example functions are implemented in AppDataProfileReporter.ts.\r\n    b) ModelMetaDataProfileReporter.ts: load a model previously trained and generated and report its parameters. Some example functions are implemented in AppModelMetaDataProfileReporter.ts.\r\n    c) ThresholdReporter.ts: load a model and a test file, then report the model performance. Some example functions are implemented in AppThresholdReporter.ts.\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./lib ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"npm run clean && npm run build && npm run doc:readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","posttest":"tslint -p test -t stylish","doc:readme":""},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-dispatcher-4.16.0-dev.20231122.4561d55.tgz","_integrity":"sha512-YT50Q6rXTLRBltVCvHgsw/dB4HPLK9zXsxFVEjOCPW+K7zHwm/AFU6xQdgk94ecry8Cv3b9rTnX3XMPVn5WW5w==","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git#master","type":"git"},"_npmVersion":"9.8.1","description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","directories":{},"_nodeVersion":"18.18.2","dependencies":{"tslib":"^1.10.0","ts-md5":"^1.2.6","argparse":"~1.0.10","readline-sync":"^1.4.10","@microsoft/bf-lu":"4.16.0-dev.20231122.4561d55"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","globby":"^10.0.1","rimraf":"^3.0.0","tslint":"^5.20.1","ts-node":"^8.4.1","typescript":"^3.7.2","@types/chai":"^4.2.4","@types/node":"^10.17.3","@types/mocha":"^5.2.7","@types/argparse":"^1.0.36"},"_npmOperationalInternal":{"tmp":"tmp/bf-dispatcher_4.16.0-dev.20231122.4561d55_1700670991919_0.6900818996610192","host":"s3://npm-registry-packages"}}},"time":{"created":"2020-08-07T23:18:42.641Z","modified":"2026-07-16T15:43:33.602Z","0.0.0":"2020-08-07T23:18:42.791Z","4.10.0-dev.20200808.5a7c973":"2020-08-08T07:18:47.254Z","4.11.0-beta.875403c":"2020-08-12T22:13:16.964Z","4.11.0-beta.6e2953b":"2020-08-12T23:59:17.420Z","4.11.0-beta.6afce3c":"2020-08-13T18:50:10.510Z","4.11.0-beta.20200818.8213b2a":"2020-08-18T20:20:32.467Z","4.11.0-beta.20200821.21f8e76":"2020-08-21T18:47:33.277Z","4.11.0-beta.20200824.6c63bf3":"2020-08-24T07:08:39.546Z","4.11.0-beta.20200904.e677869":"2020-09-04T07:09:02.131Z","4.11.0-beta.20200918.ad209cb":"2020-09-18T07:09:23.697Z","4.11.0-beta.20200919.b7fef29":"2020-09-19T07:08:16.039Z","4.11.0-beta.20200921.2320a67":"2020-09-21T07:08:35.438Z","4.11.0-beta.20200922.a071f5c":"2020-09-22T18:10:35.109Z","4.11.0-beta.20200923.7e94eb2":"2020-09-23T07:09:19.707Z","4.11.0-beta.20201003.a485403":"2020-10-03T07:07:29.773Z","4.11.0-beta.20201006.29a8107":"2020-10-06T07:08:35.832Z","4.11.0-beta.20201008.9bc8859":"2020-10-08T07:09:49.895Z","4.11.0-beta.20201009.5113321":"2020-10-09T07:09:02.792Z","4.11.0-beta.20201010.6819330":"2020-10-10T07:10:12.096Z","4.11.0-beta.20201011.d140767":"2020-10-11T07:09:04.605Z","4.11.0-beta.20201013.20d7917":"2020-10-13T07:21:53.708Z","4.11.0-beta.20201014.ce6aa6c":"2020-10-14T07:11:01.629Z","4.11.0-beta.20201015.008a3a4":"2020-10-15T07:17:24.452Z","4.11.0-beta.20201016.393c6b2":"2020-10-16T07:18:11.657Z","4.11.0-beta.20201022.4350e73":"2020-10-22T07:08:26.395Z","4.11.0-beta.20201023.67f9a0d":"2020-10-23T07:10:04.181Z","4.11.0-beta.20201024.23949ae":"2020-10-24T07:08:55.757Z","4.11.0-beta.20201025.035cdcb":"2020-10-25T07:09:19.133Z","4.11.0-beta.20201026.d8a7aef":"2020-10-26T07:09:27.845Z","4.11.0-beta.20201027.744fa02":"2020-10-27T07:09:33.052Z","4.11.0-beta.20201028.1b0cde3":"2020-10-28T07:09:33.380Z","4.11.0-beta.20201029.e864bc8":"2020-10-29T07:08:52.032Z","4.11.0-beta.20201030.881c37e":"2020-10-30T07:09:08.026Z","4.11.0-beta.20201031.17d6e9e":"2020-10-31T07:08:21.160Z","4.11.0-beta.20201101.07f38db":"2020-11-01T07:08:55.849Z","4.11.0-beta.20201103.f78e0cf":"2020-11-03T07:08:29.131Z","4.11.0-beta.20201104.673f22d":"2020-11-04T07:09:43.068Z","4.11.0-beta.20201105.0cc03df":"2020-11-05T00:14:13.361Z","4.11.0-beta.20201106.87dcae0":"2020-11-06T07:09:12.712Z","4.11.0-beta.20201107.23b4af1":"2020-11-07T07:08:23.668Z","4.11.0-beta.20201111.10062c1":"2020-11-11T07:09:31.120Z","4.11.0-beta.20201113.e85b53d":"2020-11-13T07:09:34.084Z","4.11.0-beta.20201114.d501bf0":"2020-11-14T07:09:09.924Z","4.11.0-beta.20201203.9b18c73":"2020-12-03T07:09:04.899Z","4.11.0-beta.20201203.69f8eac":"2020-12-03T18:23:13.026Z","4.11.0-beta.20201203.cd29e25":"2020-12-03T23:25:35.731Z","4.11.0-beta.20201204.cd29e25":"2020-12-04T07:09:28.596Z","4.11.0-beta.20201208.165a6d6":"2020-12-08T19:25:02.513Z","4.11.0-beta.20201209.165a6d6":"2020-12-09T07:08:42.579Z","4.11.0-beta.20201209.54c74f9":"2020-12-09T21:03:20.811Z","4.11.0-beta.20201210.9534f1b":"2020-12-10T07:09:41.556Z","4.11.0-beta.20201211.b852646":"2020-12-11T18:09:11.489Z","4.11.0-beta.20201212.7d972c3":"2020-12-12T07:09:09.583Z","4.11.0-beta.20210124.468c75a":"2021-01-24T07:08:34.807Z","4.11.0-beta.20210127.476c841":"2021-01-27T07:09:33.833Z","4.11.0-beta.20210128.3d35ae9":"2021-01-28T07:09:04.872Z","4.11.0-beta.20210129.453a0f1":"2021-01-29T07:08:58.744Z","4.11.0-beta.20210130.46cecc7":"2021-01-30T07:08:26.541Z","4.11.0-beta.20210202.f5bfbad":"2021-02-02T07:08:56.475Z","4.11.0-beta.20210203.b0a8b02":"2021-02-03T07:07:36.348Z","4.11.0-beta.20210205.12121c8":"2021-02-05T07:07:56.038Z","4.11.0-beta.20210206.dde4bda":"2021-02-06T07:08:42.433Z","4.11.0-beta.20210207.af5cc18":"2021-02-07T07:08:36.817Z","4.11.0-beta.20210209.a44ee5e":"2021-02-09T07:09:40.368Z","4.11.0-beta.20210210.116dcad":"2021-02-10T07:09:53.883Z","4.11.0-beta.20210211.c895ecf":"2021-02-11T07:09:29.188Z","4.11.0-beta.20210212.2884eed":"2021-02-12T07:09:20.355Z","4.11.0-beta.20210213.4729245":"2021-02-13T07:09:44.358Z","4.11.0-beta.20210214.41d8748":"2021-02-14T07:09:46.732Z","4.12.0-beta.20210217.668cdac":"2021-02-17T07:09:00.484Z","4.12.0-beta.20210218.668cdac":"2021-02-18T00:44:33.713Z","4.12.0-beta.20210223.d42fe4c":"2021-02-23T19:29:42.441Z","4.12.0-beta.20210224.8610f7b":"2021-02-24T07:47:14.217Z","4.12.0-beta.20210224.dd555e8":"2021-02-24T23:27:59.371Z","4.12.0-beta.20210225.6744890":"2021-02-25T07:09:36.601Z","4.12.0-beta.20210227.ba15791":"2021-02-27T07:08:57.744Z","4.12.0-beta.20210310.a728f41":"2021-03-10T17:41:14.300Z","4.12.0-beta.20210311.6018ddf":"2021-03-11T17:34:05.117Z","4.12.0-beta.20210311.98dc244":"2021-03-11T19:39:25.586Z","4.12.0-beta.20210316.cdd0819":"2021-03-16T21:07:53.681Z","4.13.0-beta.20210316.e8ec340":"2021-03-16T22:18:17.486Z","4.12.0-beta.20210322.314475a":"2021-03-22T16:36:30.450Z","4.13.0-rc0":"2021-04-02T00:50:21.962Z","4.14.0-dev.20210402.7e5fa93":"2021-04-02T07:21:14.044Z","4.14.0-dev.20210406.74baacc":"2021-04-06T07:19:50.120Z","4.14.0-dev.20210407.6745788":"2021-04-07T23:47:03.426Z","4.14.0-dev.20210408.393d33d":"2021-04-08T07:19:58.428Z","4.14.0-dev.20210408.1a5070e":"2021-04-08T17:00:36.649Z","4.14.0-dev.20210409.1a5070e":"2021-04-09T07:20:24.070Z","4.13.0-rc1":"2021-04-09T17:30:20.503Z","4.14.0-dev.20210415.161c029":"2021-04-15T07:20:07.122Z","4.14.0-dev.20210415.7e35288":"2021-04-15T19:14:52.198Z","4.14.0-dev.20210415.ab0fc09":"2021-04-15T20:23:11.118Z","4.13.0-rc2":"2021-04-15T22:50:21.010Z","4.14.0-dev.20210416.ab0fc09":"2021-04-16T07:20:09.867Z","4.14.0-dev.20210419.7257bc2":"2021-04-19T18:34:30.700Z","4.13.0":"2021-04-19T22:18:48.726Z","4.14.0-dev.20210420.fc92e9b":"2021-04-20T07:20:03.531Z","4.14.0-dev.20210422.eccdf02":"2021-04-22T07:21:33.049Z","4.13.1":"2021-04-26T21:40:38.043Z","4.14.0-dev.20210427.c0be653":"2021-04-27T07:21:20.234Z","4.13.2":"2021-04-28T22:42:00.672Z","4.14.0-dev.20210429.968ef75":"2021-04-29T15:54:57.233Z","4.13.3":"2021-04-29T20:53:07.646Z","4.14.0-dev.20210430.968ef75":"2021-04-30T07:19:15.813Z","4.14.0-dev.20210511.07bbd05":"2021-05-11T07:22:22.969Z","4.14.0-dev.20210517.4cdd4d9":"2021-05-17T07:23:20.260Z","4.14.0-dev.20210519.92dfb46":"2021-05-19T07:20:54.837Z","4.14.0-dev.20210521.64e1d40":"2021-05-21T07:22:27.633Z","4.14.0-dev.20210602.be805a8":"2021-06-02T07:25:29.731Z","4.14.0-dev.20210604.3f9ee15":"2021-06-04T07:23:21.911Z","4.14.0-dev.20210623.37f59ec":"2021-06-23T07:30:51.462Z","4.14.0-dev.20210624.eac7a60":"2021-06-24T07:22:14.298Z","4.14.0-dev.20210625.132e0cd":"2021-06-25T07:21:13.735Z","4.14.0-dev.20210629.e97e6b0":"2021-06-29T07:21:48.458Z","4.14.0-rc0":"2021-06-29T18:06:00.292Z","4.14.0-dev.20210630.8078cc6":"2021-06-30T07:21:21.866Z","4.15.0-dev.20210701.be9cd6c":"2021-07-01T07:19:30.829Z","4.15.0-dev.20210702.cbf708d":"2021-07-02T07:21:12.700Z","4.15.0-dev.20210707.cf471cf":"2021-07-07T07:20:06.198Z","4.14.0":"2021-07-07T19:26:56.472Z","4.15.0-dev.20210708.f16ca8f":"2021-07-08T07:19:12.719Z","4.14.1":"2021-07-08T18:40:31.859Z","4.15.0-dev.20210712.eb38bc0":"2021-07-12T07:19:14.003Z","4.15.0-dev.20210715.ead1751":"2021-07-15T07:19:18.662Z","4.15.0-dev.20210723.bf1f054":"2021-07-23T07:23:40.420Z","4.15.0-dev.20210803.40549ca":"2021-08-03T07:21:44.643Z","4.15.0-dev.20210805.6321dcc":"2021-08-05T07:20:14.495Z","4.15.0-dev.20210914.7a936c1":"2021-09-14T07:20:53.111Z","4.15.0-dev.20210917.b688652":"2021-09-17T07:23:39.694Z","4.15.0-dev.20210920.e4d38fc":"2021-09-20T07:23:58.122Z","4.15.0-dev.20210921.fcb2cdb":"2021-09-21T07:22:49.430Z","4.15.0-dev.20210922.c1c0528":"2021-09-22T07:22:04.429Z","4.15.0-dev.20210924.20a3f98":"2021-09-24T07:19:11.920Z","4.15.0-dev.20210928.a9a92a5":"2021-09-28T07:22:39.706Z","4.15.0-dev.20211004.edcf63e":"2021-10-04T07:25:05.227Z","4.15.0-rc0":"2021-10-15T22:52:13.627Z","4.16.0-dev.20211020.406dfef":"2021-10-20T07:23:41.831Z","4.16.0-dev.20211101.bed3c05":"2021-11-01T17:35:28.073Z","4.16.0-dev.20211102.bed3c05":"2021-11-02T07:24:36.240Z","4.16.0-dev.20211102.3f7a67a":"2021-11-02T17:48:08.044Z","4.15.0-rc1":"2021-11-02T23:17:19.535Z","4.16.0-dev.20211103.3f7a67a":"2021-11-03T07:20:12.187Z","4.16.0-dev.20211105.90655c4":"2021-11-05T07:24:46.423Z","4.15.0-rc2":"2021-11-05T18:21:20.080Z","4.15.0-rc3":"2021-11-10T23:40:15.146Z","4.16.0-dev.20211111.15b5c37":"2021-11-11T08:22:19.810Z","4.15.0":"2021-11-16T17:11:21.868Z","4.16.0-dev.20211117.19ff452":"2021-11-17T08:23:23.531Z","4.16.0-dev.20211220.6972973":"2021-12-20T08:23:29.419Z","4.16.0-dev.20220104.33b5e0a":"2022-01-04T08:23:08.686Z","4.16.0-dev.20220105.380511c":"2022-01-05T21:09:46.170Z","4.16.0-dev.20220105.c44b989":"2022-01-06T00:01:28.494Z","4.16.0-dev.20220106.c44b989":"2022-01-06T08:20:43.278Z","4.16.0-dev.20220110.b4d497d":"2022-01-10T08:21:52.963Z","4.16.0-dev.20220120.e933dfc":"2022-01-20T08:23:59.099Z","4.16.0-dev.20220228.5bb413a":"2022-02-28T08:20:31.213Z","4.16.0-dev.20220525.147c370":"2022-05-25T07:22:21.464Z","4.16.0-dev.20220526.695cee5":"2022-05-26T07:20:15.712Z","4.16.0-dev.20220725.a0ab7ca":"2022-07-25T07:21:55.216Z","4.16.0-dev.20221107.64f07e4":"2022-11-07T08:23:12.240Z","4.16.0-dev.20221111.64f07e4":"2022-11-11T23:11:50.158Z","4.16.0-dev.20221114.64f07e4":"2022-11-14T08:23:27.006Z","4.16.0-dev.20231122.4561d55":"2023-11-22T16:36:32.194Z"},"bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"author":{"name":"Microsoft"},"license":"MIT","homepage":"https://github.com/microsoft/botframework-cli","keywords":["oclif-plugin"],"repository":{"url":"https://github.com/microsoft/botframework-cli/tree/master/packages/dispatcher","type":"git"},"description":"Dispatcher contains a Softmax learner initially used for auto-active-learning down-sampling and a ML confusion-matrix evaluator on intent classification models.","maintainers":[{"email":"botframework@microsoft.com","name":"botframework"},{"email":"sgellock@microsoft.com","name":"sgellock"},{"email":"christopherwhitten@gmail.com","name":"cwhitten"},{"email":"npmjs@microsoft.com","name":"microsoft1es"},{"email":"microsoft-oss-publishing@microsoft.com","name":"microsoft-oss-releases"},{"email":"peinnes@microsoft.com","name":"peterinnesmsft"},{"email":"cmullins@gmail.com","name":"cmullins"}],"readme":"","readmeFilename":""}