andyr.jtokeniser
Class Tokeniser

java.lang.Object
  extended by andyr.jtokeniser.Tokeniser
Direct Known Subclasses:
BreakIteratorTokeniser, RegexSeparatorTokeniser, RegexTokeniser, SentenceTokeniser, StringTokeniser, WhiteSpaceTokeniser

public abstract class Tokeniser
extends java.lang.Object

Tokeniser is an abstract base class for a variety of Tokenisers.

Version:
1.2 (01-Aug-2005)
Author:
Andrew Roberts

Field Summary
protected  int currentTokenPosition
           
protected  java.util.List tokens
           
 
Constructor Summary
protected Tokeniser()
           
 
Method Summary
 int countTokens()
          Calculates the number of times that this tokeniser's nextToken method can be called before it generates an exception.
 java.util.List getTokens()
          Returns a List of string tokens.
 boolean hasMoreTokens()
          Tests if there are more tokens available from this tokeniser's string.
 java.lang.String nextToken()
          Returns the next token from this tokeniser.
 int numberOfTokens()
          Returns the total number of tokens that were extracted when tokenising the input.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

tokens

protected java.util.List tokens

currentTokenPosition

protected int currentTokenPosition
Constructor Detail

Tokeniser

protected Tokeniser()
Method Detail

countTokens

public int countTokens()
Calculates the number of times that this tokeniser's nextToken method can be called before it generates an exception.

Returns:
the number of tokens remaining in the string.

numberOfTokens

public int numberOfTokens()
Returns the total number of tokens that were extracted when tokenising the input.

Returns:
the number of tokens in the string

hasMoreTokens

public boolean hasMoreTokens()
Tests if there are more tokens available from this tokeniser's string. If this method returns true, then a subsequent call to nextToken will successfully return a token.

Returns:
true if and only if there is at least one token in the string after the current position; false otherwise.

nextToken

public java.lang.String nextToken()
                           throws java.util.NoSuchElementException
Returns the next token from this tokeniser.

Returns:
the next token from this tokeniser.
Throws:
java.util.NoSuchElementException - if there are no more tokens in this tokeniser's string.

getTokens

public java.util.List getTokens()
Returns a List of string tokens.

Returns:
a List of string tokens.