new Tokenizer(proto)
Constructs a new Tokenizer.
Parameters:
Name | Type | Description |
---|---|---|
proto |
string | Proto to tokenize |
- Source:
Members
-
index :number
-
Current index.
Type:
- number
- Source:
-
line :number
-
Current line.
Type:
- number
- Source:
-
source :string
-
Source to parse.
Type:
- string
- Source:
-
(non-null) stack :Array.<string>
-
Token stack.
Type:
- Array.<string>
- Source:
Methods
-
next() → (nullable) {string}
-
Gets the next token and advances by one.
- Source:
Returns:
Token or
null
on EOF- Type
- string
-
omit(expected) → {boolean}
-
Omits an optional token.
Parameters:
Name Type Description expected
string Expected optional token
- Source:
Returns:
true
if the token exists- Type
- boolean
-
peek() → (nullable) {string}
-
Peeks for the next token.
- Source:
Returns:
Token or
null
on EOF- Type
- string
-
skip(expected)
-
Skips a specific token and throws if it differs.
Parameters:
Name Type Description expected
string Expected token
- Source:
Throws:
-
If the actual token differs
- Type
- Error
-
toString() → {string}
-
Returns a string representation of this object.
- Source:
Returns:
String representation as of "Tokenizer(index/length)"
- Type
- string