Class TokenSample

Namespace

NOpenNLP.Tools.Tokenize

Assembly

NOpenNLP.Tools.dll

A TokenSample is text with token spans.

public class TokenSample

Inheritance

object

TokenSample

Inherited Members

object.Equals(object, object)

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

Constructors

TokenSample(IDetokenizer, string[])

public TokenSample(IDetokenizer detokenizer, string[] tokens)

Parameters

detokenizer IDetokenizer

tokens string[]

TokenSample(string, Span[])

Initializes the current instance.

public TokenSample(string text, Span[] tokenSpans)

Parameters

text string

the text which contains the tokens.

tokenSpans Span[]

the spans which mark the begin and end of the tokens.

Fields

DEFAULT_SEPARATOR_CHARS

public const string DEFAULT_SEPARATOR_CHARS = "<SPLIT>"

Field Value

string

Properties

Text

Retrieves the text.

public virtual string Text { get; }

Property Value

string

TokenSpans

Retrieves the token spans.

public virtual Span[] TokenSpans { get; }

Property Value

Span[]

Methods

Equals(object?)

Determines whether the specified object is equal to the current object.

public override bool Equals(object? obj)

Parameters

obj object

The object to compare with the current object.

Returns

bool

true if the specified object is equal to the current object; otherwise, false.

GetHashCode()

Serves as the default hash function.

public override int GetHashCode()

Returns

int

A hash code for the current object.

Parse(string, string)

public static TokenSample Parse(string sampleString, string separatorChars)

Parameters

sampleString string

separatorChars string

Returns

TokenSample

ToString()

Returns a string that represents the current object.

public override string ToString()

Returns

string

A string that represents the current object.