{"_id":"@anyhowstep/tokenizer","_rev":"4-a01175e9d2c451c89956737a0149ea7a","name":"@anyhowstep/tokenizer","description":"`npm install @anyhowstep/tokenizer`","dist-tags":{"latest":"1.1.1"},"versions":{"1.0.0":{"name":"@anyhowstep/tokenizer","version":"1.0.0","description":"`npm install @anyhowstep/tokenizer`","main":"index.js","scripts":{"test":"ts-node ./src/test/test.ts","test2":"ts-node ./src/test/test2.ts","test3":"ts-node ./src/test/test3.ts","build":"tsc -d"},"author":"","license":"ISC","dependencies":{"typescript":"^2.7.0-dev.20171220"},"devDependencies":{"@types/node":"^8.5.2","ts-node":"^4.0.2"},"gitHead":"28d1e76f78911f0773b8d911bd2a58fae8495f4d","_id":"@anyhowstep/tokenizer@1.0.0","_npmVersion":"5.3.0","_nodeVersion":"6.11.0","_npmUser":{"name":"anyhowstep","email":"justin.ng.1994@outlook.com"},"dist":{"integrity":"sha512-fwPKqGHppVF56j3EoJ5OURgAw3T306eNWkX4Go/ueNrXx4P2nxfZn7I0NW19kPTf8+J2Dq2XKVK58ELTOQxG8Q==","shasum":"f876d225de8aa45a3138a993583e74d34eb93fc2","tarball":"https://registry.npmjs.org/@anyhowstep/tokenizer/-/tokenizer-1.0.0.tgz","signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEYCIQDXQd4O4H1H2xq/E11E6/sKJ3wdyKoEP26fNo4UjnGJ8AIhAOmqq9GpWCOQQpuaiCoW7O+8AgpptXBtd8bzCYFFdhRt"}]},"maintainers":[{"name":"anyhowstep","email":"justin.ng.1994@outlook.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/tokenizer-1.0.0.tgz_1513847800621_0.49222599156200886"},"directories":{},"deprecated":"deprecated; use flavormark"},"1.1.0":{"name":"@anyhowstep/tokenizer","version":"1.1.0","description":"`npm install @anyhowstep/tokenizer`","main":"index.js","scripts":{"test":"ts-node ./src/test/test.ts","test2":"ts-node ./src/test/test2.ts","test3":"ts-node ./src/test/test3.ts","test4":"ts-node ./src/test/test4.ts","build":"tsc -d"},"author":"","license":"ISC","dependencies":{"typescript":"^2.7.0-dev.20171220"},"devDependencies":{"@types/node":"^8.5.2","ts-node":"^4.0.2"},"gitHead":"1ebb1054d2cc86442f2060094556e121463a47ac","_id":"@anyhowstep/tokenizer@1.1.0","_npmVersion":"5.3.0","_nodeVersion":"6.11.0","_npmUser":{"name":"anyhowstep","email":"justin.ng.1994@outlook.com"},"dist":{"integrity":"sha512-KkBcgcW7lx0VghIKH9o7aED6dSlByqH53259DC437IguEenFvjDCi9ItZFAtO8d9dlLx940TJ+bXIEastULQ6w==","shasum":"63be9d1d3f9811c4b141a41de79b8053aa71d6aa","tarball":"https://registry.npmjs.org/@anyhowstep/tokenizer/-/tokenizer-1.1.0.tgz","signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEUCIAxJ4LwirfZ8ZxNpqeqXg0PGoiLXJrT8DG764v0HByQbAiEAo/apALeoQKaspD90A2HwIVRcig0DrNm31IoU03nSn30="}]},"maintainers":[{"name":"anyhowstep","email":"justin.ng.1994@outlook.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/tokenizer-1.1.0.tgz_1513883228449_0.7818958605639637"},"directories":{},"deprecated":"deprecated; use flavormark"},"1.1.1":{"name":"@anyhowstep/tokenizer","version":"1.1.1","description":"`npm install @anyhowstep/tokenizer`","main":"dist/src/main/index.js","types":"dist/src/main/index.d.ts","scripts":{"test":"ts-node ./src/test/test.ts","test2":"ts-node ./src/test/test2.ts","test3":"ts-node ./src/test/test3.ts","test4":"ts-node ./src/test/test4.ts","build":"(rm -r ./dist || true) && tsc -d"},"author":"","license":"ISC","dependencies":{"typescript":"^2.7.0-dev.20171220"},"devDependencies":{"@types/node":"^8.5.2","ts-node":"^4.0.2"},"gitHead":"c27024b20261ac4270272198158b6dc850e7064c","_id":"@anyhowstep/tokenizer@1.1.1","_npmVersion":"5.3.0","_nodeVersion":"6.11.0","_npmUser":{"name":"anyhowstep","email":"justin.ng.1994@outlook.com"},"dist":{"integrity":"sha512-+fMNULp7QkDSOD97D9tEGM8dDcZBpwRWQ8BvzewxmzWFqTmIy25x2FBgON2QG3M8o3jsCr9LXVb+D0HCvEGExQ==","shasum":"7b72753e828946ab79df7f5e58cb45cc3a078c8c","tarball":"https://registry.npmjs.org/@anyhowstep/tokenizer/-/tokenizer-1.1.1.tgz","signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEUCIQDOPyE4y230doKcC9ao0SSct+FFTTxj92DBjvpM+uSPmgIgTj3yOfPBBnch46dR1KpnKR4diWbsjytam1eR3DTRuhs="}]},"maintainers":[{"name":"anyhowstep","email":"justin.ng.1994@outlook.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/tokenizer-1.1.1.tgz_1513883707864_0.39064470608718693"},"directories":{},"deprecated":"deprecated; use flavormark"}},"readme":"### Installation\n\n`npm install @anyhowstep/tokenizer`\n\n### Sample Usage\n\nSee `./src/test/test.ts` Or,\n\nSee `./src/test/test2.ts` Or,\n\nSee `./src/test/test3.ts` Or,\n\nSee `./src/test/test4.ts` Or,\n\n    import * as tokenizer from \"@anyhowstep/tokenizer\";\n\n    const src = `text goes here`;\n    const result = tokenizer.tokenize(src, (args) => {\n        {\n            const match = /^(\\s+)/.exec(args.subStr);\n            if (match != null) {\n                return {\n                    token : new tokenizer.Token({\n                        value  : match[1],\n                        type   : \"whitespace\",\n                        row    : args.row,\n                        column : args.column,\n                    })\n                };\n            }\n        }\n        {\n            const match = /^(\\S+)/.exec(args.subStr);\n            if (match != null) {\n                return {\n                    token : new tokenizer.Token({\n                        value  : match[1],\n                        type   : \"non-whitespace\",\n                        row    : args.row,\n                        column : args.column,\n                    })\n                };\n            }\n        }\n        return {\n            err : new Error(`Unknown character ${args.subStr[0]}`),\n            consumeCount : 1,\n        };\n    });\n\n    console.log(result);\n\n### TODO\n\n+ Make errors more customizable (like adding `ErrorT`)\n","maintainers":[{"name":"anyhowstep","email":"justin.ng.1994@outlook.com"}],"time":{"modified":"2022-04-04T14:27:00.910Z","created":"2017-12-21T09:16:40.690Z","1.0.0":"2017-12-21T09:16:40.690Z","1.1.0":"2017-12-21T19:07:08.605Z","1.1.1":"2017-12-21T19:15:08.113Z"},"license":"ISC","readmeFilename":"README.md"}