Old versions of JTopas
Here You can find the feature description and usage examples for the JTopas library prior to the current version. New users should always start with the most recent version, that is introduced here.
Features
These are the main characteristics of the JTopas classes and interfaces:
- one or more sorts of line and / or block comments may be added and removed during runtime,
- special sequences like operators and separators with a special meaning can be dynamically added and removed,
- support for keywords is available,
- data may be read from InputStream's as well as from other sources,
- parsing characteristics like case-sensitivity, line and column counting and whitespace handling can be set on a global as well as on a per-item base,
- read data may or may not be stored by the tokenizer,
- the specific representation of tokens may or may not be returned by the tokenizer and
- multiple tokenizers may share one data source.
There are two ways to use the JTopas classes and interfaces:
- The generic tokenizer: Use it as a black box. You need only the
de.susebox.javapackage and its subpackages. - The plugin tokenizer: Implement Your own handlers for primitive parse operations. In addition to the
de.susebox.javapackage You need also thede.susebox.jtopaspackage
While the former is very easy to use and sufficient for most situation, the latter can be used when higher performance should be achieved or a very dynamic way of token handling is nessecary. See the examples below for details.
Example 1
Here is an example Java program that extracts the contents of a HTML file using the black box approach (the generic tokenizer only). It shows, that the JTopas classes are independend from the protocol, dialect or language to be parsed. Moreover, what is extracted in which way, can be dynamically controlled. With a few alternations, for instance, it would be possible to extract all hyperlinks or the meta informations of a HTML source. There are more examples in our JUnit test cases.
// This will print the contents of a HTML file as a
// roughly formatted text
// Imports
import java.io.FileInputStream;
import java.io.InputStreamReader;
import de.susebox.java.util.Token;
import de.susebox.java.util.Tokenizer;
import de.susebox.java.util.InputStreamTokenizer;
import de.susebox.java.util.TokenizerProperty;
// class to hold main method
public class ContentsExtractor {
// Main method. Supply a HTML file name as argument
public static void main(String[] args) {
FileInputStream stream = new FileInputStream(args[0]);
InputStreamReader reader = new InputStreamReader(stream);
Tokenizer tokenizer = new InputStreamTokenizer(reader);
Token token;
int len;
int caseFlags;
// setup the tokenizer
tokenizer.setParseFlags( Tokenizer.F_NO_CASE
| Tokenizer.F_TOKEN_POS_ONLY
| Tokenizer.F_RETURN_WHITESPACES);
caseFlags = tokenizer.getParseFlags() & ~Tokenizer.F_NO_CASE;
tokenizer.setSeparators(null);
tokenizer.addBlockComment("<", ">");
tokenizer.addBlockComment("<HEAD>", "</HEAD>");
tokenizer.addBlockComment("<!--", "-->");
tokenizer.addSpecialSequence("<", "<");
tokenizer.addSpecialSequence(">", ">");
tokenizer.addSpecialSequence("ä", "ä", caseFlags);
tokenizer.addSpecialSequence("Ä", "Ä", caseFlags);
tokenizer.addSpecialSequence("ö", "ö", caseFlags);
tokenizer.addSpecialSequence("Ö", "Ö", caseFlags);
tokenizer.addSpecialSequence("ü", "ü", caseFlags);
tokenizer.addSpecialSequence("Ü", "Ü", caseFlags);
tokenizer.addSpecialSequence("ß", "ß");
tokenizer.addSpecialSequence("<b>", "");
tokenizer.addSpecialSequence("</b>", "");
tokenizer.addSpecialSequence("<i>", "");
tokenizer.addSpecialSequence("</i>", "");
tokenizer.addSpecialSequence("<code>", "");
tokenizer.addSpecialSequence("</code>", "");
// tokenize the file and print basically
// formatted context to stdout
len = 0;
while (tokenizer.hasMoreToken()) {
token = tokenizer.nextToken();
switch (token.getType()) {
case Token.NORMAL:
System.out.print(tokenizer.current());
len += token.getLength();
break;
case Token.SPECIAL_SEQUENCE:
System.out.print((String)token.getCompanion());
break;
case Token.BLOCK_COMMENT:
if (len > 0) {
System.out.println();
len = 0;
}
break;
case Token.WHITESPACE:
if (len > 75) {
System.out.println();
len = 0;
} else if (len > 0) {
System.out.print(' ');
len++;
}
break;
}
}
}
}
Example 2
The second example shows how to use the plugin tokenizer in the most straightforward way. You will notice that it is not nessecary to tell the tokenizer, which character combinations comrpise comments, special sequences or separators (although it is possible). Instead a handler-like pattern is applied with some primitive parse methods.
// Imports
import java.io.FileInputStream;
import java.io.InputStreamReader;
import java.util.ArrayList;
import de.susebox.java.util.Token;
import de.susebox.java.util.Tokenizer;
import de.susebox.java.util.TokenizerProperty;
import de.susebox.jtopas.PluginTokenizer;
import de.susebox.jtopas.InputStreamSource;
import de.susebox.jtopas.SequenceHandler;
import de.susebox.jtopas.SeparatorHandler;
// Plugin class
public class XMLTokenizer
implements SequenceHandler, SeparatorHandler
{
// Tokenizer properties can be defined as constants
private static final Object START_TAG_COMP = new Object;
private static final Object END_TAG_COMP = new Object;
private static final TokenizerProperty STRING_PROP
= new TokenizerProperty(Token.STRING,
new String[] { "\"", "\"", "\\" },
null );
private static final TokenizerProperty COMMENT_PROP
= new TokenizerProperty(Token.BLOCK_COMMENT,
new String[] { "<!--", "-->" },
null );
private static final TokenizerProperty SPEC_COMMENT_PROP
= new TokenizerProperty(Token.SPECIAL_SEQUENCE,
new String[] { "<!" },
null );
private static final TokenizerProperty END_TAG
= new TokenizerProperty(Token.SPECIAL_SEQUENCE,
new String[] { "</" },
END_TAG_COMP );
private static final TokenizerProperty START_TAG
= new TokenizerProperty(Token.SPECIAL_SEQUENCE,
new String[] { "<" },
START_TAG_COMP );
private static final TokenizerProperty TAG_END
= new TokenizerProperty(Token.SPECIAL_SEQUENCE,
new String[] { ">" },
null );
// member variables
private PluginTokenizer _tokenizer = null;
private ArrayList _tagList = new ArrayList(1024);;
// Main method. Supply a XML file name as argument
public static void main(String[] args) {
FileInputStream stream = new FileInputStream(args[0]);
InputStreamReader reader = new InputStreamReader(stream);
PluginTokenizer tokenizer = new PluginTokenizer();
// setup the tokenizer
tokenizer.setSource(new InputStreamSource(reader));
tokenizer.setParseFlags( Tokenizer.F_TOKEN_POS_ONLY
| Tokenizer.F_RETURN_WHITESPACES);
tokenizer.setSequenceHandler(this);
tokenizer.setSeparatorHandler(this);
// get all the XML tags
while (tokenizer.hasMoreToken()) {
Token token = tokenizer.nextToken();
if (token.getType() == Token.SPECIAL_SEQUENCE) {
if (token.getCompanion() == START_TAG_COMP) {
token = tokenizer.nextToken();
_tagList.add(new TagInfo(tokenizer.current(),
START_TAG_COMP));
} else if (token.getCompanion() == END_TAG_COMP) {
token = tokenizer.nextToken();
_tagList.add(new TagInfo(tokenizer.current(),
END_TAG_COMP));
}
}
}
}
// Plugin interface method
public void setTokenizer(PluginTokenizer tokenizer) {
_tokenizer = tokenizer;
}
// SeparatorHandler interface method
public boolean isSeparator(char testChar) {
return testChar == '=';
}
// get the maximum possible length for a special sequence
public int getSequenceMaxLength() {
return 4; // length of "<!--"
}
// The sequence handler
public TokenizerProperty isSequenceCommentOrString(
int startAt, int maxChars
)
{
TokenizerProperty prop = null;
switch (_tokenizer.getCharUnchecked(startingAtPos)) {
case '"':
// strings are attribute values in XML
prop = STRING_PROP;
break;
case '<':
// tag opening
switch (_tokenizer.getCharUnchecked(startAt + 1)) {
case '!':
if ( _tokenizer.getCharUnchecked(startAt + 2) == '-'
&& _tokenizer.getCharUnchecked(startAt + 3) == '-') {
prop = COMMENT_PROP;
} else {
prop = SPEC_COMMENT_PROP;
}
break;
case '/':
prop = END_TAG;
break;
default:
prop = START_TAG;
break;
}
break;
case '>':
// tag closing
prop = TAG_END;
break;
}
}
}