Class DictionaryDetokenizer

Namespace

NOpenNLP.Tools.Tokenize

Assembly

NOpenNLP.Tools.dll

A rule based detokenizer. Simple rules which indicate in which direction a token should be moved are looked up in a DetokenizationDictionary object.

public class DictionaryDetokenizer : IDetokenizer

Inheritance

object

DictionaryDetokenizer

Implements

IDetokenizer

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Remarks

IDetokenizer DetokenizationDictionary

Constructors

DictionaryDetokenizer(DetokenizationDictionary)

A rule based detokenizer. Simple rules which indicate in which direction a token should be moved are looked up in a DetokenizationDictionary object.

public DictionaryDetokenizer(DetokenizationDictionary dict)

Parameters

dict DetokenizationDictionary

Remarks

IDetokenizer DetokenizationDictionary

Methods

Detokenize(string[])

Detokenize the input tokens.

public virtual DetokenizationOperation[] Detokenize(string[] tokens)

Parameters

tokens string[]

the tokens to detokenize.

Returns

DetokenizationOperation[]

the merge operations to detokenize the input tokens.

Detokenize(string[], string?)

Detokenize the input tokens into a String. Tokens which are connected without a space inbetween can be separated by a split marker.

public virtual string Detokenize(string[] tokens, string? splitMarker)

Parameters

tokens string[]

the token which should be concatenated

splitMarker string

the split marker or null

Returns

string

the concatenated tokens