# Class: Tokenizer

## ProtoBuf.DotProto. Tokenizer

prototype tokenizer

#### new Tokenizer(proto)

  Constructs a new Tokenizer.

  ##### Parameters:

  Name | Type | Description
  --- | --- | ---
  `proto` | string | Proto to tokenize

  Source:
    - ProtoBuf.js, line 495

### Members

#### index :number

  Current index.

  ##### Type:

  - number
  Source:
    - ProtoBuf.js, line 509

#### line :number

  Current line.

  ##### Type:

  - number
  Source:
    - ProtoBuf.js, line 516

#### source :string

  Source to parse.

  ##### Type:

  - string
  Source:
    - ProtoBuf.js, line 502

#### (non-null) stack :Array.<string>

  Token stack.

  ##### Type:

  - Array.<string>
  Source:
    - ProtoBuf.js, line 523

### Methods

#### next() → (nullable) {string}

  Gets the next token and advances by one.

  Source:
    - ProtoBuf.js, line 563

  ##### Returns:

  Token or `null` on EOF

  Type
    string

#### omit(expected) → {boolean}

  Omits an optional token.

  ##### Parameters:

  Name | Type | Description
  --- | --- | ---
  `expected` | string | Expected optional token

  Source:
    - ProtoBuf.js, line 658

  ##### Returns:

  `true` if the token exists

  Type
    boolean

#### peek() → (nullable) {string}

  Peeks for the next token.

  Source:
    - ProtoBuf.js, line 632

  ##### Returns:

  Token or `null` on EOF

  Type
    string

#### skip(expected)

  Skips a specific token and throws if it differs.

  ##### Parameters:

  Name | Type | Description
  --- | --- | ---
  `expected` | string | Expected token

  Source:
    - ProtoBuf.js, line 647

  ##### Throws:

  If the actual token differs
  Type
    Error

#### toString() → {string}

  Returns a string representation of this object.

  Source:
    - ProtoBuf.js, line 671

  ##### Returns:

  String representation as of "Tokenizer(index/length)"

  Type
    string