Class AbstractTokenizer

Namespace

NOpenNLP.Tools.Tokenize

Assembly

NOpenNLP.Tools.dll

public abstract class AbstractTokenizer : ITokenizer

Inheritance

object

AbstractTokenizer

Implements

ITokenizer

Derived

SimpleTokenizer

TokenizerME

WhitespaceTokenizer

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Methods

Tokenize(string)

Splits a string into its atomic parts

public virtual string[] Tokenize(string s)

Parameters

s string

The string to be tokenized.

Returns

string[]

The String[] with the individual tokens as the array elements.

TokenizePos(string)

Finds the boundaries of atomic parts in a string.

public abstract Span[] TokenizePos(string s)

Parameters

s string

The string to be tokenized.

Returns

Span[]

The Span[] with the spans (offsets into s) for each token as the individuals array elements.