Class TokSpanEventStream

Namespace

NOpenNLP.Tools.Tokenize

Assembly

NOpenNLP.Tools.dll

This class reads the TokenSamples from the given IObjectStream<T> and converts the TokenSamples into Events which can be used by the maxent library for training.

public class TokSpanEventStream : AbstractEventStream<TokenSample>, IObjectStream<Event?>, IDisposable

Inheritance

object

ObjectStreamBase<Event>

AbstractEventStream<TokenSample>

TokSpanEventStream

Implements

IObjectStream<Event>

IDisposable

Inherited Members

AbstractEventStream<TokenSample>.Read()

AbstractEventStream<TokenSample>.Reset()

AbstractEventStream<TokenSample>.Dispose(bool)

ObjectStreamBase<Event>.Dispose()

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Constructors

TokSpanEventStream(IObjectStream<TokenSample?>, bool)

Initializes the current instance.

public TokSpanEventStream(IObjectStream<TokenSample?> tokenSamples, bool skipAlphaNumerics)

Parameters

tokenSamples IObjectStream<TokenSample>

the samples to create events from

skipAlphaNumerics bool

if true alpha numerics are skipped

TokSpanEventStream(IObjectStream<TokenSample?>, bool, ITokenContextGenerator)

Initializes the current instance.

public TokSpanEventStream(IObjectStream<TokenSample?> tokenSamples, bool skipAlphaNumerics, ITokenContextGenerator cg)

Parameters

tokenSamples IObjectStream<TokenSample>

the samples to create events from

skipAlphaNumerics bool

if true alpha numerics are skipped

cg ITokenContextGenerator

the context generator

TokSpanEventStream(IObjectStream<TokenSample?>, bool, Regex, ITokenContextGenerator)

Initializes the current instance.

public TokSpanEventStream(IObjectStream<TokenSample?> tokenSamples, bool skipAlphaNumerics, Regex alphaNumeric, ITokenContextGenerator cg)

Parameters

tokenSamples IObjectStream<TokenSample>

the samples to create events from

skipAlphaNumerics bool

if true alpha numerics are skipped

alphaNumeric Regex

the alpha numeric pattern

cg ITokenContextGenerator

the context generator

Methods

CreateEvents(TokenSample)

Adds training events to the event stream for each of the specified tokens.

protected override IEnumerable<Event> CreateEvents(TokenSample tokenSample)

Parameters

tokenSample TokenSample

character offsets into the specified text.

Returns

IEnumerable<Event>

The text of the tokens.