Class TokenizerFactory

Namespace

NOpenNLP.Tools.Tokenize

Assembly

NOpenNLP.Tools.dll

The factory that provides ITokenizer default implementations and resources. Users can extend this class if their application requires overriding the ITokenContextGenerator, Dictionary etc.

public class TokenizerFactory : BaseToolFactory

Inheritance

object

BaseToolFactory

TokenizerFactory

Inherited Members

BaseToolFactory.artifactProvider

BaseToolFactory.Init(IArtifactProvider)

BaseToolFactory.CreateArtifactSerializersMap()

BaseToolFactory.Create(string, IArtifactProvider)

BaseToolFactory.Create(Type, IArtifactProvider)

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Constructors

TokenizerFactory()

Creates a TokenizerFactory that provides the default implementation of the resources.

public TokenizerFactory()

TokenizerFactory(string, Dictionary?, bool, Regex)

Creates a TokenizerFactory. Use this constructor to programmatically create a factory.

public TokenizerFactory(string languageCode, Dictionary? abbreviationDictionary, bool useAlphaNumericOptimization, Regex alphaNumericPattern)

Parameters

languageCode string

the language of the natural text

abbreviationDictionary Dictionary

an abbreviations dictionary

useAlphaNumericOptimization bool

if true alpha numerics are skipped

alphaNumericPattern Regex

null or a custom alphanumeric pattern (default is: "^[A-Za-z0-9]+$", provided by DEFAULT_ALPHANUMERIC

Properties

AbbreviationDictionary

Gets the abbreviation dictionary

public virtual Dictionary? AbbreviationDictionary { get; }

Property Value

Dictionary

null or the abbreviation dictionary

AlphaNumericPattern

Gets the alpha numeric pattern.

public virtual Regex? AlphaNumericPattern { get; }

Property Value

Regex

the user specified alpha numeric pattern or a default.

ContextGenerator

Gets the context generator

public virtual ITokenContextGenerator ContextGenerator { get; }

Property Value

ITokenContextGenerator

a new instance of the context generator

LanguageCode

Retrieves the language code.

public virtual string? LanguageCode { get; }

Property Value

string

the language code

UseAlphaNumericOptmization

Gets whether to use alphanumeric optimization.

public virtual bool UseAlphaNumericOptmization { get; }

Property Value

bool

true if the alpha numeric optimization is enabled, otherwise false

Methods

Create(string?, string, Dictionary?, bool, Regex)

Factory method the framework uses create a new TokenizerFactory.

public static TokenizerFactory? Create(string? subclassName, string languageCode, Dictionary? abbreviationDictionary, bool useAlphaNumericOptimization, Regex alphaNumericPattern)

Parameters

subclassName string

the name of the class implementing the TokenizerFactory

languageCode string

the language code the tokenizer should use

abbreviationDictionary Dictionary

an optional dictionary containing abbreviations, or null if not present

useAlphaNumericOptimization bool

indicate if the alpha numeric optimization should be enabled or disabled

alphaNumericPattern Regex

the pattern the alpha numeric optimization should use

Returns

TokenizerFactory

the instance of the ITokenizer Factory

Exceptions

InvalidFormatException

if once of the input parameters doesn't comply if the expected format

CreateArtifactMap()

Creates an IDictionary<TKey, TValue> with pairs of keys and objects. The models implementation should call this constructor that creates a model programmatically.

The base implementation will return a Dictionary<TKey, TValue> that should be populated by sub-classes.

public override IDictionary<string, object> CreateArtifactMap()

Returns

IDictionary<string, object>

CreateManifestEntries()

Creates the manifest entries that will be added to the model manifest

public override IDictionary<string, string> CreateManifestEntries()

Returns

IDictionary<string, string>

the manifest entries to added to the model manifest

Init(string, Dictionary?, bool, Regex)

protected virtual void Init(string languageCode, Dictionary? abbreviationDictionary, bool useAlphaNumericOptimization, Regex alphaNumericPattern)

Parameters

languageCode string

abbreviationDictionary Dictionary

useAlphaNumericOptimization bool

alphaNumericPattern Regex

ValidateArtifactMap()

Validates the parsed artifacts. If something is not valid subclasses should throw an InvalidFormatException.

Note: Subclasses should generally invoke super.validateArtifactMap at the beginning of this method.

public override void ValidateArtifactMap()

Exceptions

InvalidFormatException