Class SimpleTokenizer

Namespace

NOpenNLP.Tools.Tokenize

Assembly

NOpenNLP.Tools.dll

Performs tokenization using character classes.

public class SimpleTokenizer : AbstractTokenizer, ITokenizer

Inheritance

object

AbstractTokenizer

SimpleTokenizer

Implements

ITokenizer

Inherited Members

AbstractTokenizer.Tokenize(string)

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Constructors

SimpleTokenizer()

public SimpleTokenizer()

Remarks

Deprecated: Use INSTANCE field instead to obtain an instance, constructor will be made private in the future.

Fields

INSTANCE

public static readonly SimpleTokenizer INSTANCE

Field Value

SimpleTokenizer

Methods

TokenizePos(string)

Finds the boundaries of atomic parts in a string.

public override Span[] TokenizePos(string s)

Parameters

s string

The string to be tokenized.

Returns

Span[]

The Span[] with the spans (offsets into s) for each token as the individuals array elements.