Interface IDetokenizer

Namespace

NOpenNLP.Tools.Tokenize

Assembly

NOpenNLP.Tools.dll

A Detokenizer merges tokens back to their untokenized representation.

public interface IDetokenizer

Methods

Detokenize(string[])

Detokenize the input tokens.

DetokenizationOperation[] Detokenize(string[] tokens)

Parameters

tokens string[]

the tokens to detokenize.

Returns

DetokenizationOperation[]

the merge operations to detokenize the input tokens.

Detokenize(string[], string?)

Detokenize the input tokens into a String. Tokens which are connected without a space inbetween can be separated by a split marker.

string Detokenize(string[] tokens, string? splitMarker)

Parameters

tokens string[]

the token which should be concatenated

splitMarker string

the split marker or null

Returns

string

the concatenated tokens