Class TokenizerME

Namespace: NOpenNLP.Tools.Tokenize

Assembly: NOpenNLP.Tools.dll

A ITokenizer for converting raw text into separated tokens. It uses Maximum Entropy to make its decisions. The features are loosely based off of Jeff Reynar's UPenn thesis "Topic Segmentation: Algorithms and Applications.", which is available from his homepage: http://www.cis.upenn.edu/~jcreynar.

This tokenizer needs a statistical model to tokenize a text which reproduces the tokenization observed in the training data used to create the model. The TokenizerModel class encapsulates the model and provides methods to create it from the binary representation.

A tokenizer instance is not thread safe. For each thread one tokenizer must be instantiated which can share one TokenizerModel instance to safe memory.

To train a new model the Train(IObjectStream<TokenSample?>, TokenizerFactory, TrainingParameters) method can be used.

Sample usage:

Stream modelIn;

...

TokenizerModel model = new TokenizerModel(modelIn);

ITokenizer tokenizer = new TokenizerME(model);

string[] tokens = tokenizer.Tokenize("A sentence to be tokenized.");

public class TokenizerME : AbstractTokenizer, ITokenizer

Inheritance: object

AbstractTokenizer

TokenizerME

Implements: ITokenizer

Inherited Members: AbstractTokenizer.Tokenize(string)

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Remarks

ITokenizer TokenizerModel See TokenSample.

Constructors

TokenizerME(TokenizerModel)

public TokenizerME(TokenizerModel model)

Parameters

model TokenizerModel

TokenizerME(TokenizerModel, Factory)

public TokenizerME(TokenizerModel model, Factory factory)

Parameters

model TokenizerModel
factory Factory

Remarks

Deprecated: Use TokenizerFactory to extend the ITokenizer functionality

Fields

NO_SPLIT

Constant indicates no token split.

public const string NO_SPLIT = "F"

Field Value

string

SPLIT

Constant indicates a token split.

public const string SPLIT = "T"

Field Value

string

alphaNumeric

Alpha-Numeric Regex

public static readonly Regex alphaNumeric

Field Value

Regex

Remarks

Deprecated: As of release 1.5.2, replaced by GetAlphanumeric(string?)

Properties

TokenProbabilities

Returns the probabilities associated with the most recent calls to Tokenize(string) or TokenizePos(string).

public virtual double[] TokenProbabilities { get; }

Property Value

double[]: probability for each token returned for the most recent call to tokenize. If not applicable an empty array is returned.

UseAlphaNumericOptimization

Returns the value of the alpha-numeric optimization flag.

public virtual bool UseAlphaNumericOptimization { get; }

Property Value

bool: true if the tokenizer should use alpha-numeric optimization, false otherwise.

Methods

TokenizePos(string)

Tokenizes the string.

public override Span[] TokenizePos(string d)

Parameters

d string: The string to be tokenized.

Returns

Span[]: A span array containing individual tokens as elements.

Train(IObjectStream<TokenSample?>, TokenizerFactory, TrainingParameters)

Trains a model for the TokenizerME.

public static TokenizerModel Train(IObjectStream<TokenSample?> samples, TokenizerFactory factory, TrainingParameters mlParams)

Parameters

samples IObjectStream<TokenSample>: the samples used for the training.
factory TokenizerFactory: a TokenizerFactory to get resources from
mlParams TrainingParameters: the machine learning train parameters

Returns

TokenizerModel: the trained TokenizerModel

Exceptions

IOException: it throws an IOException if an IOException is thrown during IO operations on a temp file which is created during training. Or if reading from the IObjectStream<T> fails.