Class TokenizerModel

Namespace

NOpenNLP.Tools.Tokenize

Assembly

NOpenNLP.Tools.dll

The TokenizerModel is the model used by a learnable ITokenizer.

public sealed class TokenizerModel : BaseModel, IArtifactProvider

Inheritance

object

BaseModel

TokenizerModel

Implements

IArtifactProvider

Inherited Members

BaseModel.TRAINING_CUTOFF_PROPERTY

BaseModel.TRAINING_ITERATIONS_PROPERTY

BaseModel.TRAINING_EVENTHASH_PROPERTY

BaseModel.toolFactory

BaseModel.GetArtifactSerializer(string)

BaseModel.CreateArtifactSerializers()

BaseModel.CreateArtifactSerializers(IDictionary<string, IArtifactSerializer>)

BaseModel.GetManifestProperty(string)

BaseModel.Language

BaseModel.Version

BaseModel.Serialize(Stream)

BaseModel.Serialize(FileInfo)

BaseModel.Serialize(string)

BaseModel.GetArtifact<T>(string)

BaseModel.IsLoadedFromSerialized()

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.ReferenceEquals(object, object)

object.ToString()

Constructors

TokenizerModel(IMaxentModel, IDictionary<string, string>?, TokenizerFactory)

Initializes the current instance.

public TokenizerModel(IMaxentModel tokenizerModel, IDictionary<string, string>? manifestInfoEntries, TokenizerFactory tokenizerFactory)

Parameters

tokenizerModel IMaxentModel

the model

manifestInfoEntries IDictionary<string, string>

the manifest

tokenizerFactory TokenizerFactory

the factory

TokenizerModel(FileInfo)

Initializes the current instance.

public TokenizerModel(FileInfo modelFile)

Parameters

modelFile FileInfo

the file containing the tokenizer model

Exceptions

IOException

if reading from the stream fails in anyway

TokenizerModel(Stream)

Initializes the current instance.

public TokenizerModel(Stream @in)

Parameters

in Stream

the Input Stream to load the model from

Exceptions

IOException

if reading from the stream fails in anyway

InvalidFormatException

if the stream doesn't have the expected format

Properties

Abbreviations

public Dictionary? Abbreviations { get; }

Property Value

Dictionary

DefaultFactory

Sub-classes should override this method if their module has a default BaseToolFactory sub-class.

protected override Type DefaultFactory { get; }

Property Value

Type

the default BaseToolFactory for the module, or null if none.

Factory

public TokenizerFactory Factory { get; }

Property Value

TokenizerFactory

MaxentModel

public IMaxentModel MaxentModel { get; }

Property Value

IMaxentModel

UseAlphaNumericOptimization

public bool UseAlphaNumericOptimization { get; }

Property Value

bool

Methods

ValidateArtifactMap()

Validates the parsed artifacts. If something is not valid subclasses should throw an InvalidFormatException.

Note: Subclasses should generally invoke super.validateArtifactMap at the beginning of this method.

protected override void ValidateArtifactMap()

Exceptions

InvalidFormatException

See Also

TokenizerME