Class StringUtil

Namespace: NOpenNLP.Tools.Util

Assembly: NOpenNLP.Tools.dll

public class StringUtil

Inheritance: object

StringUtil

Inherited Members: object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Methods

ComputeShortestEditScript(string, string, int[,], StringBuffer)

Computes the Shortest Edit Script (SES) to convert a word into its lemma. This is based on Chrupala's PhD thesis (2008).

public static void ComputeShortestEditScript(string wordForm, string lemma, int[,] distance, StringBuffer permutations)

Parameters

wordForm string: the token
lemma string: the target lemma
distance int[,]: the levenshtein distance
permutations StringBuffer: the number of permutations

DecodeShortestEditScript(string, string)

Read predicted SES by the lemmatizer model and apply the permutations to obtain the lemma from the wordForm.

public static string DecodeShortestEditScript(string wordForm, string permutations)

Parameters

wordForm string: the wordForm
permutations string: the permutations predicted by the lemmatizer model

Returns

string: the lemma

GetShortestEditScript(string, string)

Get the SES required to go from a word to a lemma.

public static string GetShortestEditScript(string wordForm, string lemma)

Parameters

wordForm string: the word
lemma string: the lemma

Returns

string: the shortest edit script

IsEmpty(string)

Returns true if Length is 0 or null.

public static bool IsEmpty(string theString)

Parameters

theString string

Returns

bool: true if Length is 0, otherwise false

Remarks

Since 1.5.1.

IsWhitespace(char)

Determines if the specified character is a whitespace.

A character is considered a whitespace when one of the following conditions is meet:

Its a char.IsWhiteSpace(char) whitespace.
Its a part of the Unicode Zs category (SpaceSeparator).

Character.isWhitespace(int) does not include no-break spaces. In OpenNLP no-break spaces are also considered as white spaces.

public static bool IsWhitespace(char charCode)

Parameters

charCode char

Returns

bool: true if white space otherwise false

IsWhitespace(int)

Determines if the specified character is a whitespace.

A character is considered a whitespace when one of the following conditions is meet:

Its a char.IsWhiteSpace(char) whitespace.
Its a part of the Unicode Zs category (SpaceSeparator).

Character.isWhitespace(int) does not include no-break spaces. In OpenNLP no-break spaces are also considered as white spaces.

public static bool IsWhitespace(int charCode)

Parameters

charCode int

Returns

bool: true if white space otherwise false

LevenshteinDistance(string, string)

Computes the Levenshtein distance of two strings in a matrix. Based on pseudo-code provided here: https://en.wikipedia.org/wiki/Levenshtein_distance#Computing_Levenshtein_distance which in turn is based on the paper Wagner, Robert A.; Fischer, Michael J. (1974), "The String-to-String Correction Problem", Journal of the ACM 21 (1): 168-173

public static int[,] LevenshteinDistance(string wordForm, string lemma)

Parameters

wordForm string: the form
lemma string: the lemma

Returns

int[,]: the distance

SplitDroppingTrailingEmpty(string, char)

Splits value on separator the way Java's String.split does, discarding trailing empty strings.

public static string[] SplitDroppingTrailingEmpty(string value, char separator)

Parameters

value string
separator char

Returns

string[]

Remarks

Authored for NOpenNLP; not part of the Apache OpenNLP source. Java's String.split(regex) uses a limit of zero, which discards all trailing empty strings; .NET's Split(params char[]) keeps them. So "a b ".split(" ") is 3 elements in Java and 4 in .NET, the last empty. That difference is silent and changes results: an extra empty token reaches a model, or an empty class name reaches an extension loader. Interior empty strings are kept by both, and both yield a single-element array when the separator is absent.

The empty input is the one case where Java does not drop the trailing empty: "".split(",") is one element, the empty string, while ",".split(",") is none. Both were checked against a real JDK. That single element matters -- a caller that reads the last element of the result, as the OntoNotes line splitter does, would otherwise fail on an empty document where upstream succeeds.

ToLowerCase(string)

Converts to lower case independent of the current locale via J2N.Character.ToLowerInvariant(int) which uses mapping information from the UnicodeData file.

public static string ToLowerCase(string @string)

Parameters

string string

Returns

string: lower cased String

ToUpperCase(string)

Converts to upper case independent of the current locale via char.ToUpperInvariant(char) which uses mapping information from the UnicodeData file.

public static string ToUpperCase(string @string)

Parameters

string string

Returns

string: upper cased String