Class TokenizerStream

Namespace

NOpenNLP.Tools.Tokenize

Assembly

NOpenNLP.Tools.dll

The TokenizerStream uses a tokenizer to tokenize the input string and output TokenSamples.

public class TokenizerStream : ObjectStreamBase<TokenSample?>, IObjectStream<TokenSample?>, IDisposable

Inheritance

object

ObjectStreamBase<TokenSample>

TokenizerStream

Implements

IObjectStream<TokenSample>

IDisposable

Inherited Members

ObjectStreamBase<TokenSample>.Dispose()

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Constructors

TokenizerStream(ITokenizer, IObjectStream<string?>)

The TokenizerStream uses a tokenizer to tokenize the input string and output TokenSamples.

public TokenizerStream(ITokenizer tokenizer, IObjectStream<string?> input)

Parameters

tokenizer ITokenizer

input IObjectStream<string>

Methods

Dispose(bool)

Releases the resources from this object stream.

protected override void Dispose(bool disposing)

Parameters

disposing bool

Read()

Returns the next object. Calling this method repeatedly until it returns null will return each object from the underlying source exactly once.

public override TokenSample? Read()

Returns

TokenSample

the next object or null to signal that the stream is exhausted

Exceptions

IOException

if there is an error during reading

IOException

if there is an error during reading

Reset()

Repositions the stream at the beginning and the previously seen object sequence will be repeated exactly. This method can be used to re-read the stream if multiple passes over the objects are required.

The implementation of this method is optional.

public override void Reset()

Exceptions

IOException

if there is an error during resetting the stream

NotSupportedException

if reset is not supported on this stream

NotSupportedException

if reset is not supported on the input stream