Enum DetokenizationOperation

Namespace

NOpenNLP.Tools.Tokenize

Assembly

NOpenNLP.Tools.dll

This enum contains an operation for every token to merge the tokens together to their detokenized form.

public enum DetokenizationOperation

Fields

MergeToRight = 0

The current token should be attached to the begin token on the right side.

MergeToLeft = 1

The current token should be attached to the string on the left side.

MergeBoth = 2

The current token should be attached to the string on the left side, as well as to the begin token on the right side.

NoOperation = 3

Do not perform a merge operation for this token, but is possible that another token can be attached to the left or right side of this one.

Remarks

NOpenNLP: upstream declares this enum nested inside the Detokenizer interface. C# does not allow type declarations inside an interface, so it is hoisted to the namespace level.