com.topologi.diffx.load.text
Interface TextTokenizer

All Known Implementing Classes:

TokenizerByChar, TokenizerByText, TokenizerByWord

public interface TextTokenizer

An interface for text tokenizers.

Text tokenisers are used to return a list of TextEvent from a piece of text.

Version:

3 February 2005

Author:

Christophe Lauret

Method Summary

TextGranularity	granurality() Returns the text granularity of this tokenizer.
List<TextEvent>	tokenize(CharSequence seq) Returns the list of TextEvent corresponding to the specified character sequence.

Method Detail

tokenize

List<TextEvent> tokenize(CharSequence seq)

Returns the list of TextEvent corresponding to the specified character sequence.

Parameters:

seq - the character sequence to tokenize.

Returns:

the corresponding list.

granurality

TextGranularity granurality()

Returns the text granularity of this tokenizer.

Returns:

the text granularity of this tokenizer.