Options
All
  • Public
  • Public/Protected
  • All
Menu

Hierarchy

Implements

Index

Constructors

constructor

Properties

Protected Optional _POSTAG

_POSTAG: POSTAG

Protected Optional _TABLE

_TABLE: any

Optional inited

inited: boolean

name

name: string

Optional priority

priority: number

segment

segment: Segment

type

type: "tokenizer" = "tokenizer"

Static type

type: "tokenizer" = "tokenizer"

Methods

Protected _cache

  • _cache(...argv: any[]): void

Protected _splitUnknow

  • _splitUnknow<T, U>(words: T[], fn: function): U[]

Protected _splitUnset

  • _splitUnset<T, U>(words: T[], fn: function): U[]

Protected createToken

  • createToken<T, U>(data: T, skipCheck?: boolean, attr?: U & IWordDebugInfo): T

Protected debugToken

  • debugToken<T, U>(data: T, attr?: U & IWordDebugInfo, returnToken?: true, ...argv: any[]): T

init

  • init(segment: Segment, ...argv: any[]): this

Protected sliceToken

  • sliceToken<T>(words: T[], pos: number, len: number, data: T, skipCheck?: boolean): T[]

split

splitForeign

  • splitForeign(text: string, cur?: number): IWord[]
  • 匹配包含的英文字符和数字,并分割

    Parameters

    • text: string

      文本

    • Optional cur: number

      开始位置

    Returns IWord[]

    返回格式 {w: '单词', c: 开始位置}

Static Protected _init

Static init

  • init<T>(segment: Segment, ...argv: any[]): T

Generated using TypeDoc