{"_id":"@coder-ka/ll-parsing","_rev":"4-2291523e2262c8030af8890349516d60","name":"@coder-ka/ll-parsing","dist-tags":{"latest":"1.0.3"},"versions":{"1.0.0":{"name":"@coder-ka/ll-parsing","version":"1.0.0","keywords":[],"author":"","license":"MIT","_id":"@coder-ka/ll-parsing@1.0.0","maintainers":[{"name":"coder-ka","email":"coder.ka.issues@gmail.com"}],"homepage":"https://github.com/coder-ka/ll-parsing#readme","bugs":{"url":"https://github.com/coder-ka/ll-parsing/issues"},"dist":{"shasum":"3af935ad116c0fb0b87a59ae492dd036db1d5974","tarball":"https://registry.npmjs.org/@coder-ka/ll-parsing/-/ll-parsing-1.0.0.tgz","fileCount":6,"integrity":"sha512-J/Dq9idI5TNat+A9x21uIU63LoUZ29S+T/zL6I9poaBIq1qNpipt/EE/mHo5suj89V0GTjNoR+vGiRZo6raBoA==","signatures":[{"sig":"MEUCIF6ZASxje0dzjkGHylj8B7E67mQfTUPNZrrRs86Iz4xFAiEA8+fonwOM+Mry6cqNYIQGL+xFi+3/gBVdHTaB5Rxot3I=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":14342},"main":"./dist/index.cjs","exports":{".":{"types":"./types/index.d.ts","import":"./dist/index.mjs","require":"./dist/index.cjs"}},"scripts":{"test":"tsx tests","build":"tsc && npm run build:esm && npm run build:cjs","watch":"concurrently \"tsc --watch\" \"npm run build:esm -- --watch\" \"npm run build:cjs -- --watch\"","build:cjs":"esbuild src/index.ts --bundle --format=cjs --outfile=dist/index.cjs","build:esm":"esbuild src/index.ts --bundle --format=esm --outfile=dist/index.mjs","build:node":"tsc && npm run build:node:esm && npm run build:node:cjs","watch:node":"concurrently \"tsc --watch\" \"npm run build:node:esm -- --watch\" \"npm run build:node:cjs -- --watch\"","build:node:cjs":"npm run build:cjs -- --platform=node","build:node:esm":"npm run build:esm -- --platform=node"},"_npmUser":{"name":"coder-ka","email":"coder.ka.issues@gmail.com"},"repository":{"url":"git+https://github.com/coder-ka/ll-parsing.git","type":"git"},"_npmVersion":"9.9.3","description":"","directories":{},"_nodeVersion":"20.15.0","_hasShrinkwrap":false,"devDependencies":{"tsx":"^4.9.3","assert":"^2.1.0","esbuild":"^0.20.2","typescript":"^5.4.5","@types/node":"^22.5.0","concurrently":"^8.2.2","@types/assert":"^1.5.10","@coder-ka/testing":"^1.0.1"},"_npmOperationalInternal":{"tmp":"tmp/ll-parsing_1.0.0_1730746470259_0.6364499151209695","host":"s3://npm-registry-packages"}},"1.0.1":{"name":"@coder-ka/ll-parsing","version":"1.0.1","keywords":[],"author":"","license":"MIT","_id":"@coder-ka/ll-parsing@1.0.1","maintainers":[{"name":"coder-ka","email":"coder.ka.issues@gmail.com"}],"homepage":"https://github.com/coder-ka/ll-parsing#readme","bugs":{"url":"https://github.com/coder-ka/ll-parsing/issues"},"dist":{"shasum":"095011cbbd623e935bdd213dc2c3fa7978391a6e","tarball":"https://registry.npmjs.org/@coder-ka/ll-parsing/-/ll-parsing-1.0.1.tgz","fileCount":5,"integrity":"sha512-HxZi1K4WZjdvFdEgvzecrjc0atzZkqStFkXuVyoUVenrng8YFBvdqK8uRdQtnNpQjvcHnFK59WR6s1yAYr9gPQ==","signatures":[{"sig":"MEQCIGw0wThqGH4/zXYi+X6JyQDb2qhG+hYrnAO7NRMW5GfrAiAIyxwTqKH6vuCMfYdJaVIuWNxJ/U+u7OGFYbLWxeHesA==","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":12507},"main":"./dist/index.cjs","exports":{".":{"types":"./types/index.d.ts","import":"./dist/index.mjs","require":"./dist/index.cjs"}},"scripts":{"test":"tsx tests","build":"tsc && npm run build:esm && npm run build:cjs","watch":"concurrently \"tsc --watch\" \"npm run build:esm -- --watch\" \"npm run build:cjs -- --watch\"","build:cjs":"esbuild src/index.ts --bundle --format=cjs --outfile=dist/index.cjs","build:esm":"esbuild src/index.ts --bundle --format=esm --outfile=dist/index.mjs","build:node":"tsc && npm run build:node:esm && npm run build:node:cjs","watch:node":"concurrently \"tsc --watch\" \"npm run build:node:esm -- --watch\" \"npm run build:node:cjs -- --watch\"","build:node:cjs":"npm run build:cjs -- --platform=node","build:node:esm":"npm run build:esm -- --platform=node"},"_npmUser":{"name":"coder-ka","email":"coder.ka.issues@gmail.com"},"repository":{"url":"git+https://github.com/coder-ka/ll-parsing.git","type":"git"},"_npmVersion":"9.9.3","description":"","directories":{},"_nodeVersion":"22.17.1","_hasShrinkwrap":false,"devDependencies":{"tsx":"^4.9.3","assert":"^2.1.0","esbuild":"^0.20.2","typescript":"^5.4.5","@types/node":"^22.5.0","concurrently":"^8.2.2","@types/assert":"^1.5.10","@coder-ka/testing":"^1.0.1"},"_npmOperationalInternal":{"tmp":"tmp/ll-parsing_1.0.1_1755243324354_0.2804706518219471","host":"s3://npm-registry-packages-npm-production"}},"1.0.2":{"name":"@coder-ka/ll-parsing","version":"1.0.2","keywords":[],"author":"","license":"MIT","_id":"@coder-ka/ll-parsing@1.0.2","maintainers":[{"name":"coder-ka","email":"coder.ka.issues@gmail.com"}],"homepage":"https://github.com/coder-ka/ll-parsing#readme","bugs":{"url":"https://github.com/coder-ka/ll-parsing/issues"},"dist":{"shasum":"0e67fbd38e4e66e7a7a7d67e5435927c5a2e6b67","tarball":"https://registry.npmjs.org/@coder-ka/ll-parsing/-/ll-parsing-1.0.2.tgz","fileCount":5,"integrity":"sha512-zgu/q1roq7B08+I6RPRA86py0WmEePSAq2y1qZfotRo8GUX5CXTrVLRDcKil407iORp7MloBcw6zxfe3UJNuLg==","signatures":[{"sig":"MEUCIQDUKFnQqxox6EdU7XHxNstqquiTPJfQp27nuiELjPHuaAIgAfQvsfw1n2YWd+DhZ7CPxsXnkmfNVnoN9G0BouAMFxY=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":24481},"main":"./dist/index.cjs","exports":{".":{"types":"./types/index.d.ts","import":"./dist/index.mjs","require":"./dist/index.cjs"}},"scripts":{"test":"tsx --test tests/**/*.test.ts","build":"tsc && npm run build:esm && npm run build:cjs","watch":"concurrently \"tsc --watch\" \"npm run build:esm -- --watch\" \"npm run build:cjs -- --watch\"","build:cjs":"esbuild src/index.ts --bundle --format=cjs --outfile=dist/index.cjs","build:esm":"esbuild src/index.ts --bundle --format=esm --outfile=dist/index.mjs","build:node":"tsc && npm run build:node:esm && npm run build:node:cjs","watch:node":"concurrently \"tsc --watch\" \"npm run build:node:esm -- --watch\" \"npm run build:node:cjs -- --watch\"","build:node:cjs":"npm run build:cjs -- --platform=node","build:node:esm":"npm run build:esm -- --platform=node"},"_npmUser":{"name":"coder-ka","email":"coder.ka.issues@gmail.com"},"repository":{"url":"git+https://github.com/coder-ka/ll-parsing.git","type":"git"},"_npmVersion":"9.9.3","description":"> [!NOTE] > This README has been machine-translated. For the original Japanese version, please refer to [README_ja.md](./README_ja.md).","directories":{},"_nodeVersion":"22.17.1","_hasShrinkwrap":false,"devDependencies":{"tsx":"^4.9.3","assert":"^2.1.0","esbuild":"^0.20.2","typescript":"^5.4.5","@types/node":"^22.5.0","concurrently":"^8.2.2","@types/assert":"^1.5.10"},"_npmOperationalInternal":{"tmp":"tmp/ll-parsing_1.0.2_1773316083584_0.15678032080682636","host":"s3://npm-registry-packages-npm-production"}},"1.0.3":{"name":"@coder-ka/ll-parsing","version":"1.0.3","description":"ll-parsing is a lightweight library for creating LL(k) parsers.","main":"./dist/index.cjs","exports":{".":{"import":"./dist/index.mjs","require":"./dist/index.cjs","types":"./types/index.d.ts"}},"scripts":{"build:cjs":"esbuild src/index.ts --bundle --format=cjs --outfile=dist/index.cjs","build:esm":"esbuild src/index.ts --bundle --format=esm --outfile=dist/index.mjs","build":"tsc && npm run build:esm && npm run build:cjs","build:node:cjs":"npm run build:cjs -- --platform=node","build:node:esm":"npm run build:esm -- --platform=node","build:node":"tsc && npm run build:node:esm && npm run build:node:cjs","watch":"concurrently \"tsc --watch\" \"npm run build:esm -- --watch\" \"npm run build:cjs -- --watch\"","watch:node":"concurrently \"tsc --watch\" \"npm run build:node:esm -- --watch\" \"npm run build:node:cjs -- --watch\"","test":"tsx --test tests/**/*.test.ts"},"repository":{"type":"git","url":"git+https://github.com/coder-ka/ll-parsing.git"},"keywords":[],"author":"","license":"MIT","bugs":{"url":"https://github.com/coder-ka/ll-parsing/issues"},"homepage":"https://github.com/coder-ka/ll-parsing#readme","devDependencies":{"@types/assert":"^1.5.10","@types/node":"^22.5.0","assert":"^2.1.0","concurrently":"^8.2.2","esbuild":"^0.20.2","tsx":"^4.9.3","typescript":"^5.4.5"},"_id":"@coder-ka/ll-parsing@1.0.3","_nodeVersion":"22.17.1","_npmVersion":"9.9.3","dist":{"integrity":"sha512-tigIb1jf85maxYe1GkBxWFcYgn/bRbYsQr1xm6q7LMygpYY9Ex18cJqMHwSgY1vlBiYvg+tUUJv6OP6ytQ/G5w==","shasum":"e94c24c67d440536c7120c99e2ec2a4cc6b01e8b","tarball":"https://registry.npmjs.org/@coder-ka/ll-parsing/-/ll-parsing-1.0.3.tgz","fileCount":5,"unpackedSize":24508,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEUCIBTSyLWKHt1r17twwgU2Hpp0SYAQq168ewj0uhG9tgbdAiEAsz91FicPJX6/izvLQeZJ62U87THYEOkmmPHO9KMD0uw="}]},"_npmUser":{"name":"coder-ka","email":"coder.ka.issues@gmail.com"},"directories":{},"maintainers":[{"name":"coder-ka","email":"coder.ka.issues@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/ll-parsing_1.0.3_1773335842777_0.9468188148711751"},"_hasShrinkwrap":false}},"time":{"created":"2024-11-04T18:54:30.164Z","modified":"2026-03-12T17:17:23.077Z","1.0.0":"2024-11-04T18:54:30.504Z","1.0.1":"2025-08-15T07:35:24.561Z","1.0.2":"2026-03-12T11:48:03.717Z","1.0.3":"2026-03-12T17:17:22.946Z"},"bugs":{"url":"https://github.com/coder-ka/ll-parsing/issues"},"license":"MIT","homepage":"https://github.com/coder-ka/ll-parsing#readme","keywords":[],"repository":{"type":"git","url":"git+https://github.com/coder-ka/ll-parsing.git"},"description":"ll-parsing is a lightweight library for creating LL(k) parsers.","maintainers":[{"name":"coder-ka","email":"coder.ka.issues@gmail.com"}],"readme":"# ll-parsing\n\n> [!NOTE]\n> This README has been machine-translated. For the original Japanese version, please refer to [README_ja.md](./README_ja.md).\n\nll-parsing is a lightweight library for creating LL(k) parsers.\n\n## Features\n\n- **Lightweight**: Implemented with simple loops instead of recursive calls to ensure lightweight operation.\n- **Streaming Support**: Does not strain memory; can take any stream that produces strings as input.\n- **Flexible**: Allows for flexible conversion processes and error handling, making it suitable for purposes beyond AST generation.\n- **Zero Dependencies**: Fast installation and more secure.\n- **Simple Implementation**: The core functionality is around 150 lines, and even with utility functions, it's about 200 lines—shorter than this document.\n\n## Installation\n\n```bash\nnpm install @coder-ka/ll-parsing\n```\n\n## Getting Started\n\nThe following is an example of parsing an IPv4 address.\n\n```ts\nimport { createLLParser, parseError, createSimpleLexer } from \"@coder-ka/ll-parsing\";\n\nconst ipv4SegmentRegex = /\\d+/;\nconst S = Symbol(\"S\");\nconst SEG = Symbol(\"SEG\");\nconst DOT = Symbol(\"DOT\")\nconst $ = Symbol(\"$\");\nconst ipv4Parser = createLLParser<{ value: string[] }>(\n  {\n    [S]() {\n      return [SEG, DOT, SEG, DOT, SEG, DOT, SEG]\n    },\n    [SEG]: ([token], { index, line, inlineIndex }, state) => {\n      if (ipv4SegmentRegex.test(token)) {\n        state.value.push(token);\n        return [token];\n      } else {\n        return [\n          parseError({\n            message: `Unexpected token: '${token}'.`,\n            token,\n            index,\n            line,\n            inlineIndex,\n          }),\n        ];\n      }\n    },\n    [DOT]: ([token], { index, line, inlineIndex }) => {\n      if (token === \".\") {\n        return [token];\n      } else {\n        return [\n          parseError({\n            message: `Unexpected token: '${token}'.`,\n            token,\n            index,\n            line,\n            inlineIndex,\n          }),\n        ];\n      }\n    },\n  },\n  () => [S, $]\n);\n\nconst lexer = createSimpleLexer({\n  separatorRegex: /\\./,\n})\n\nconst inputBuffer = lexer(async function* () {\n  yield \"127.0.0.1\";\n}());\n\n\nconst parsed = await ipv4Parser.parse(\n  inputBuffer,\n  {\n    value: []\n  }\n);\n\nconsole.log(parsed.state.value); // [\"127\", \"0\", \"0\", \"1\"]\n```\n\nIt might seem like overkill for just an IPv4 address, but by understanding the basics through this code, you'll be able to implement parsers for much larger codebases concisely.\n\nFirst, as a pre-parsing step, you need to convert the string stream into a token stream using a lexer.\n\nIn this case, we pass the input string stream to the lexer as follows:\n\n```ts\nconst inputBuffer = lexer(async function* () {\n  yield \"127.0.0.1\";\n}());\n```\n\nThe lexer is created using the `createSimpleLexer` utility function like this:\n\n```ts\nconst lexer = createSimpleLexer({\n  separatorRegex: /\\./,\n})\n```\n\nThis lexer generates a token stream like the following (it actually includes line and inline index information):\n\n```ts\n[\n    [\"127\"],\n    [\".\"],\n    [\"0\"],\n    [\".\"],\n    [\"0\"],\n    [\".\"],\n    [\"1\"],\n]\n```\n\nSince each element is an array of length 1, this is an LL(1) parser.\n\nWhile details are provided later, you can also support LL(k) by implementing your own lexer.\n\nWith the input stream created, defined the parser using a stack and a rule table.\n\nThe following code shows the stack initialization:\n\n```ts\n() => [S, $]\n```\n\n`S` is the start symbol, and `$` is the end symbol. Parsing is successful if the input stream is fully consumed while only `$` remains on the stack.\n\nThe rule table is an object whose keys are symbol values representing markers and whose values are functions.\n\nIn the previous code, symbols are defined as follows:\n\n```ts\nconst S = Symbol(\"S\");\nconst SEG = Symbol(\"SEG\");\nconst DOT = Symbol(\"DOT\")\nconst $ = Symbol(\"$\");\n```\n\nNow, let's look at the rule table using these as keys.\n\nThe parsing process proceeds by looking at the stack array from the front. If a symbol value is present, it is removed from the stack, the corresponding function in the rule table is executed, and the return value is pushed onto the top of the stack.\n\nWith the stack initialized as mentioned above, the function corresponding to the start symbol `S` is called first.\n\n```ts\n{\n  [S]() {\n    return [SEG, DOT, SEG, DOT, SEG, DOT, SEG]\n  },\n}\n```\n\nAfter this function executes, the stack state transitions to:\n\n```\n[SEG, DOT, SEG, DOT, SEG, DOT, SEG, $]\n```\n\nAgain, since there's a symbol `SEG` at the top of the stack, a state transition using the rule table occurs.\n\n```ts\n{\n    [SEG]: ([token], { index, line, inlineIndex }, state) => {\n      if (ipv4SegmentRegex.test(token)) {\n        state.value.push(token);\n        return [token];\n      } else {\n        return [\n          parseError({\n            message: `Unexpected token: '${token}'.`,\n            token,\n            index,\n            line,\n            inlineIndex,\n          }),\n        ];\n      }\n    },\n}\n```\n\nThe symbol `SEG` corresponds to each segment (numeric part) of the IPv4 address.\n\nTherefore, it first validates whether the current token is a numeric part using `ipv4SegmentRegex` (`/\\d+/`).\n\nIf validation succeeds, it adds the token to the parsing state array `state.value` and returns an array.\n\nAs mentioned earlier, the returned array is pushed onto the top of the stack.\n\nSo, it transitions to the following state:\n\n```\n[\"127\", DOT, SEG, DOT, SEG, DOT, SEG, $]\n```\n\nThis time, a string instead of a symbol value is at the top of the stack.\n\nIn this case, the parser consumes a token matching the string from the input stream and removes the string from the stack.\n\nThat is, the current value of the input stream and the stack will be:\n\n```\nStack: [DOT, SEG, DOT, SEG, DOT, SEG, $]\nCurrent input stream value: [\".\"]\n```\n\nThis process repeats until the input stream is fully consumed, at which point parsing ends.\n\nThe state is stored in the return value of the `parse` method.\n\n```ts\nparsed.stack // [$]\nparsed.state // { value: [\"127\", \"0\", \"0\", \"1\"] }\nparsed.errors // []\n```\n\nIt might seem complex, but you can see a definite pattern.\n\nFirst, the function for the start symbol indicates that an IPv4 address is a fixed sequence of symbols shown in the return array.\n\n```ts\n{\n  [S]() {\n    return [SEG, DOT, SEG, DOT, SEG, DOT, SEG]\n  },\n}\n```\n\nAnd when validating a token and advancing the input stream, you need to return the token string.\n\n```ts\n{\n    [SEG]: ([token], { index, line, inlineIndex }, state) => {\n      if (ipv4SegmentRegex.test(token)) {\n        state.value.push(token);\n        return [token];\n      } else {\n        // ...omitted\n      }\n    },\n}\n```\n\nIf the next symbol changes depending on the token, you might return something like `[token, SOME_SYMBOL]`.\n\nThe key is that you can advance the input stream with strings, and the symbol values on the stack represent what symbol should come next.\n\nThus, since the start symbol `S` function knows the exact sequence of symbols for an IPv4 address, it could push all symbols onto the stack at once, and subsequent processes only needed to perform string checks and token consumption.\n\nParsing more complex languages requires more processing, but since it's proportional to the rules and the number of lookahead tokens (i.e., *k*) rather than the scale or complexity of the code itself, simple languages lead to simple implementations.\n\nFor more specific examples, please refer to the [ontype implementation](https://github.com/coder-ka/ontype/blob/main/src/index.ts).\n\n## API\n\nThe main APIs in ll-parsing are the `createLLParser` function and the `parse` method of the object it returns.\n\nAdditionally, the `createSimpleLexer` utility function is provided.\n\nLet's look at each.\n\n### `createLLParser` Function\n\n`createLLParser` is a function that creates an LL parser, which proceeds completely synchronously.\n\nAs seen in the earlier example, it takes an object (the table) where symbol values are keys and functions are values as its first argument, and it initializes the stack with its second argument.\n\nThe returned parser object has only the `parse` method.\n\nThus, the signature can be simply expressed as:\n\n```\ncreateLLParser(rules, initStack): { parse }\n```\n\nFirst, `initStack` is a function that returns the initial value of the stack.\n\n```ts\ncreateLLParser(\n  rules,\n  () => [S, $]\n)\n```\n\nThe stack is just an array, and in the above example, it contains two symbols: `S` and `$`.\n\n`S` and `$` are not special symbols provided by ll-parsing; they are defined by the user.\n\nUsually, you'll start with these two, but there might be cases where you restore from a saved state.\n\nNext, let's look at `rules`.\n\n```ts\ncreateLLParser(\n  rules: {\n    [S]([token], { index, line, inlineIndex }, state) {\n      return [];\n    }\n  },\n  () => [S, $],\n)\n```\n\nAs explained earlier, `rules` is an object.\n\nKeys are symbol values representing markers.\n\nValues are functions where the first argument is an array of lookahead tokens (for LL(1), an array of length 1), the second argument is information about the token's position in the source, and the third argument is a state object for including information such as an AST.\n\nThe number of lookahead tokens is determined by the lexer implementation. Lexers are explained later.\n\nInformation about the position in the source includes these three, each starting from 0:\n\n- `index`: The index of the token within the source, including newlines.\n- `line`: The line position of the token within the source.\n- `inlineIndex`: The index within the line where the token is located.\n\nFor example, the `return` token in the following source:\n\n```\nfn add(a, b) {\n  return a + b\n}\n```\n\nhas the following position information:\n\n```json\n{\n  \"index\": 17,\n  \"line\": 1,\n  \"inlineIndex\": 2\n}\n```\n\nThese are useful for error reporting.\n\nThe state object can be any shape you choose. For example, it can be `{ ast: YourLangAST }`.\n\nNext, let's look at the `parse` method of the returned parser object.\n\n#### `parse` Method\n\n`createLLParser` creates a parser object. The parser object has only the `parse` method.\n\nThe `parse` method performs actual parsing based on the table and initial stack specified in the `createLLParser` arguments.\n\nHere is a simplified signature:\n\n```ts\nconst { parse } = createLLParser(...);\n\nparse(lexed, state, options);\n```\n\n`lexed` (the first argument) is the object output by the lexer. Details are provided later.\n\nThe second argument is the initial value of the state object. As mentioned before, the contents of the state object are modified during parsing.\n\nThe third argument is options for adjusting behavior. Its signature is:\n\n```ts\ntype ParseOptions = {\n  onError: \"stop\" | \"throw\" | \"continue\";\n  debug?: boolean;\n}\n```\n\nThe `onError` option can take three values:\n\n- `stop`: Halts parsing without throwing an exception if an error occurs (default).\n- `throw`: Throws an exception if an error occurs.\n- `continue`: Continues processing even if an error occurs.\n\nThe `debug` option, if specified, outputs useful information for debugging during parsing.\n\nThe `parse` method returns:\n\n```ts\nPromise<{\n  stack: TStack;\n  errors: ParseError[];\n  state: TState;\n  index: number;\n}>\n```\n\n- `stack`: The final stack.\n- `errors`: A list of errors that occurred. If the `onError` option is not `continue`, there will be at most one.\n- `state`: The state object.\n- `index`: The final index position.\n\nYou can use this information to see if parsing succeeded or to retrieve the AST.\n\nFor example, you can check whether only the end symbol remains in the `stack`, whether `errors` is empty, and whether `index` matches the length of the source code.\n\nNext, let's look at the lexer.\n\n### `createSimpleLexer` Function\n\nFirst, let me explain what a lexer is.\n\nA lexer in ll-parsing is a function that creates an async generator that `yield`s data related to tokens.\n\nFor example:\n\n```ts\nconst lexer = (async function* () {\n  yield {\n    tokens: [\"a\"],\n    token: \"a\",\n    index: 1,\n    line: 0,\n    inlineIndex: 1,\n  };\n  yield {\n    tokens: [\"b\"],\n    token: \"b\",\n    index: 2,\n    line: 0,\n    inlineIndex: 2,\n  };\n  yield {\n    tokens: [\"c\"],\n    token: \"c\",\n    index: 3,\n    line: 0,\n    inlineIndex: 3,\n  };\n});\n```\n\nWhile hardcoded here, the source imagined from this code would be the string `abc`.\n\nThe type of object that can be `yield`ed is:\n\n```ts\ntype LexedItem = {\n  tokens: string[];\n  index: number;\n  line: number;\n  inlineIndex: number;\n}\n```\n\n- `tokens`: An array of strings. The length of this array corresponds to the lookahead number *k*.\n- `index`: The index indicating the token's position.\n- `line`: The line the token belongs to.\n- `inlineIndex`: The index within the line.\n\nThe `parse` method takes the async generator as its first argument.\n\n```ts\nparse(\n  lexer(),\n  {...}\n)\n```\n\nWhile you can implement a lexer however you like, for LL(1) where the input is a string stream, which covers most cases, the `createSimpleLexer` function is helpful.\n\nHere is a simplified signature:\n\n```ts\nconst lexer = createSimpleLexer({\n  separatorRegex,\n  newlineRegex,\n})\n```\n\n- `separatorRegex`: A regular expression to identify separator characters.\n- `newlineRegex`: A regular expression representing newlines. The default is `/^\\r?\\n$/`.\n\nThe separator characters themselves are also streamed as tokens.\n\nFor example:\n\n```ts\nconst lexer = createSimpleLexer({\n  separatorRegex: /\\./,\n})\n\nconst inputBuffer = lexer(async function* () {\n  yield \"127.0.0.1\";\n}());\n```\n\nwill stream tokens like this:\n\n```ts\n[\n  [\"127\"],\n  [\".\"],\n  [\"0\"],\n  [\".\"],\n  [\"0\"],\n  [\".\"],\n  [\"1\"],\n]\n```\n\nFor most common LL(1) languages, `createSimpleLexer` should provide the desired tokenization.\n\n## License\n\nMIT\n","readmeFilename":"README.md"}