andyr.jtokeniser
Class WhiteSpaceTokeniser
java.lang.Object
andyr.jtokeniser.Tokeniser
andyr.jtokeniser.WhiteSpaceTokeniser
public class WhiteSpaceTokeniser
- extends Tokeniser
The WhiteSpaceTokeniser class a basic tokeniser that uses whitespace to separate tokens from the input string.
Whitespace characters are "\t\n\r\f"
: the space character, the
tab character, the newline character, the carriage-return character,
and the form-feed character.
The following is one example of the use of the tokenizer. The code:
WhiteSpaceTokeniser wst = new WhiteSpaceTokeniser("the cat sat on the mat");
while (wst.hasMoreTokens()) {
System.out.println(wst.nextToken());
}
prints the following output:
the
sat
on
the
mat
- Version:
- 1.2 (01-Aug-2005)
- Author:
- Andrew Roberts
Constructor Summary |
WhiteSpaceTokeniser(java.lang.String input)
Creates a WhiteSpaceTokeniser that tokenises the input. |
Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
WhiteSpaceTokeniser
public WhiteSpaceTokeniser(java.lang.String input)
- Creates a WhiteSpaceTokeniser that tokenises the input. The tokeniser
uses the default delimiter set, which is
"\t\n\r\f"
: the space character, the
tab character, the newline character, the carriage-return character,
and the form-feed character. Delimiter characters themselves will
not be treated as tokens.
- Parameters:
input
- a string from which the tokens will be extracted.