Class WhitespaceTokenizer

Namespace

NOpenNLP.Tools.Tokenize

Assembly

NOpenNLP.Tools.dll

This tokenizer uses white spaces to tokenize the input text.

To obtain an instance of this tokenizer use the static final INSTANCE field.

public class WhitespaceTokenizer : AbstractTokenizer, ITokenizer

Inheritance

object

AbstractTokenizer

WhitespaceTokenizer

Implements

ITokenizer

Inherited Members

AbstractTokenizer.Tokenize(string)

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Fields

INSTANCE

Use this static reference to retrieve an instance of the WhitespaceTokenizer.

public static readonly WhitespaceTokenizer INSTANCE

Field Value

WhitespaceTokenizer

Methods

TokenizePos(string)

Finds the boundaries of atomic parts in a string.

public override Span[] TokenizePos(string d)

Parameters

d string

Returns

Span[]

The Span[] with the spans (offsets into s) for each token as the individuals array elements.