Namespace NOpenNLP.Tools.Tokenize

Contains classes related to finding token or words in a string. All tokenizer implement the ITokenizer interface. Currently there is the learnable TokenizerME, the WhitespaceTokenizer and the SimpleTokenizer which is a character class tokenizer.

Namespaces

NOpenNLP.Tools.Tokenize.Lang

Classes

AbstractTokenizer

DefaultTokenContextGenerator

Generate events for maxent decisions for tokenization.

DetokenizationDictionary

DetokenizationOperationTypeExtensions

Extension methods for DetokenizationOperationType.

DetokenizerEvaluator

The DetokenizerEvaluator measures the performance of the given IDetokenizer with the provided reference TokenSamples.

DictionaryDetokenizer

A rule based detokenizer. Simple rules which indicate in which direction a token should be moved are looked up in a DetokenizationDictionary object.

SimpleTokenizer

Performs tokenization using character classes.

TokSpanEventStream

This class reads the TokenSamples from the given IObjectStream<T> and converts the TokenSamples into Events which can be used by the maxent library for training.

TokenSample

A TokenSample is text with token spans.

TokenSampleStream

This class is a stream filter which reads in string encoded samples and creates TokenSamples out of them. The input string sample is tokenized if a whitespace or the special separator chars occur.

Sample:
"token1 token2 token3<SPLIT>token4"
The tokens token1 and token2 are separated by a whitespace, token3 and token3 are separated by the special character sequence, in this case the default split sequence.

The sequence must be unique in the input string and is not escaped.

TokenizerCrossValidator

TokenizerEvaluator

The TokenizerEvaluator measures the performance of the given ITokenizer with the provided reference TokenSamples.

TokenizerFactory

The factory that provides ITokenizer default implementations and resources. Users can extend this class if their application requires overriding the ITokenContextGenerator, Dictionary etc.

TokenizerME

A ITokenizer for converting raw text into separated tokens. It uses Maximum Entropy to make its decisions. The features are loosely based off of Jeff Reynar's UPenn thesis "Topic Segmentation: Algorithms and Applications.", which is available from his homepage: http://www.cis.upenn.edu/~jcreynar.

This tokenizer needs a statistical model to tokenize a text which reproduces the tokenization observed in the training data used to create the model. The TokenizerModel class encapsulates the model and provides methods to create it from the binary representation.

A tokenizer instance is not thread safe. For each thread one tokenizer must be instantiated which can share one TokenizerModel instance to safe memory.

To train a new model the Train(IObjectStream<TokenSample?>, TokenizerFactory, TrainingParameters) method can be used.

Sample usage:

Stream modelIn;

...

TokenizerModel model = new TokenizerModel(modelIn);

ITokenizer tokenizer = new TokenizerME(model);

string[] tokens = tokenizer.Tokenize("A sentence to be tokenized.");

TokenizerModel

The TokenizerModel is the model used by a learnable ITokenizer.

TokenizerStream

The TokenizerStream uses a tokenizer to tokenize the input string and output TokenSamples.

WhitespaceTokenStream

This stream formats a TokenSamples into whitespace separated token strings.

WhitespaceTokenizer

This tokenizer uses white spaces to tokenize the input text.

To obtain an instance of this tokenizer use the static final INSTANCE field.

Interfaces

IDetokenizer

A Detokenizer merges tokens back to their untokenized representation.

ITokenContextGenerator

Interface for TokenizerME context generators.

ITokenizer

The interface for tokenizers, which segment a string into its tokens.

Tokenization is a necessary step before more complex NLP tasks can be applied, these usually process text on a token level. The quality of tokenization is important because it influences the performance of high-level task applied to it.

In segmented languages like English most words are segmented by white spaces expect for punctuations, etc. which is directly attached to the word without a white space in between, it is not possible to just split at all punctuations because in abbreviations dots are a part of the token itself. A tokenizer is now responsible to split these tokens correctly.

In non-segmented languages like Chinese tokenization is more difficult since words are not segmented by a whitespace.

Tokenizers can also be used to segment already identified tokens further into more atomic parts to get a deeper understanding. This approach helps more complex task to gain insight into tokens which do not represent words like numbers, units or tokens which are part of a special notation.

For most further task it is desirable to over tokenize rather than under tokenize.

ITokenizerEvaluationMonitor

Enums

DetokenizationOperation

This enum contains an operation for every token to merge the tokens together to their detokenized form.

DetokenizationOperationType

Specifies in which direction a token should be moved when detokenizing.