Class TokenSampleStream

Namespace

NOpenNLP.Tools.Tokenize

Assembly

NOpenNLP.Tools.dll

This class is a stream filter which reads in string encoded samples and creates TokenSamples out of them. The input string sample is tokenized if a whitespace or the special separator chars occur.

Sample:
"token1 token2 token3<SPLIT>token4"
The tokens token1 and token2 are separated by a whitespace, token3 and token3 are separated by the special character sequence, in this case the default split sequence.

The sequence must be unique in the input string and is not escaped.

public class TokenSampleStream : FilterObjectStream<string?, TokenSample?>, IObjectStream<TokenSample?>, IDisposable

Inheritance

object

ObjectStreamBase<TokenSample>

FilterObjectStream<string, TokenSample>

TokenSampleStream

Implements

IObjectStream<TokenSample>

IDisposable

Inherited Members

FilterObjectStream<string, TokenSample>.samples

FilterObjectStream<string, TokenSample>.Reset()

FilterObjectStream<string, TokenSample>.Dispose(bool)

ObjectStreamBase<TokenSample>.Reset()

ObjectStreamBase<TokenSample>.Dispose(bool)

ObjectStreamBase<TokenSample>.Dispose()

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Constructors

TokenSampleStream(IObjectStream<string?>, string)

This class is a stream filter which reads in string encoded samples and creates TokenSamples out of them. The input string sample is tokenized if a whitespace or the special separator chars occur.

Sample:
"token1 token2 token3<SPLIT>token4"
The tokens token1 and token2 are separated by a whitespace, token3 and token3 are separated by the special character sequence, in this case the default split sequence.

The sequence must be unique in the input string and is not escaped.

public TokenSampleStream(IObjectStream<string?> sampleStrings, string separatorChars = "<SPLIT>")

Parameters

sampleStrings IObjectStream<string>

separatorChars string

Methods

Read()

Returns the next object. Calling this method repeatedly until it returns null will return each object from the underlying source exactly once.

public override TokenSample? Read()

Returns

TokenSample

the next object or null to signal that the stream is exhausted

Exceptions

IOException

if there is an error during reading

IOException

if there is an error during reading