|
As the title indicates, this is software written in Java and designed for being used in Java applications, libraries and other Java environments. In this version, the following features are provided:
This list will be frequently extended by new features, implementations of standard Java API's, and, last but not least, extensions to existing features.
Currently, JTopas consists of:
NOTE: The java package root is "de.susebox" according to our registered internet domain, not jtopas or similar.
|
|
If You need to parse more sophisticated texts than can easily be handled using java.util.StringTokenizer or java.io.StreamTokenizer, and You dont want to use JavaCC or JTB or other parser generators, our generic, multi-purpose tokenizer may be just right. It has more abilities to parse source code, HTML etc. than the mentioned JDK classes without the complexity and power of parser generators. In fact, with a more or less simple wrapper it can be used as a tokenizer for JavaCC and JTB, if the generated tokenizers do not fit for some reason.
In the following example, we extract the contents from a HTML file to produce a roughly formatted text file. A more sophisticated version can be found in one of the JUnit tests (see junit/de/susebox/java/util/TestInputStreamTokenizer.java, Method testContentsFormatting):
// Imports
import java.io.FileInputStream;
import java.io.InputStreamReader;
import de.susebox.java.util.Token;
import de.susebox.java.util.Tokenizer;
import de.susebox.java.util.InputStreamTokenizer;
import de.susebox.java.util.TokenizerProperty;
// class to hold main method
public class ContentsExtractor {
// Main method. Supply a HTML file name as argument
public static void main(String[] args) {
FileInputStream stream = new FileInputStream(args[0]);
InputStreamReader reader = new InputStreamReader(stream);
Tokenizer tokenizer = new InputStreamTokenizer(reader);
Token token;
int len;
int caseFlags;
// setup the tokenizer
tokenizer.setParseFlags( Tokenizer.F_NO_CASE
| Tokenizer.F_TOKEN_POS_ONLY
| Tokenizer.F_RETURN_WHITESPACES);
caseFlags = tokenizer.getParseFlags()
& ~Tokenizer.F_NO_CASE;
tokenizer.setSeparators(null);
tokenizer.addBlockComment("<", ">");
tokenizer.addBlockComment("<HEAD>", "</HEAD>");
tokenizer.addBlockComment("<!--", "-->");
tokenizer.addSpecialSequence("<", "<");
tokenizer.addSpecialSequence(">", ">");
tokenizer.addSpecialSequence("ä", "ä", caseFlags);
tokenizer.addSpecialSequence("Ä", "Ä", caseFlags);
tokenizer.addSpecialSequence("ö", "ö", caseFlags);
tokenizer.addSpecialSequence("Ö", "Ö", caseFlags);
tokenizer.addSpecialSequence("ü", "ü", caseFlags);
tokenizer.addSpecialSequence("Ü", "Ü", caseFlags);
tokenizer.addSpecialSequence("<b>", "");
tokenizer.addSpecialSequence("</b>", "");
tokenizer.addSpecialSequence("<i>", "");
tokenizer.addSpecialSequence("</i>", "");
tokenizer.addSpecialSequence("<code>", "");
tokenizer.addSpecialSequence("</code>", "");
// tokenize the file and print basically
// formatted context to stdout
len = 0;
while (tokenizer.hasMoreToken()) {
token = tokenizer.nextToken();
switch (token.getType()) {
case Token.NORMAL:
System.out.print(tokenizer.current());
len += token.getLength();
break;
case Token.SPECIAL_SEQUENCE:
System.out.print((String)token.getCompanion());
break;
case Token.BLOCK_COMMENT:
if (len > 0) {
System.out.println();
len = 0;
}
break;
case Token.WHITESPACE:
if (len > 75) {
System.out.println();
len = 0;
} else if (len > 0) {
System.out.print(' ');
len++;
}
break;
}
}
}
}
The generic tokenizer supports the following features, some of them can be found in the example above:
- support for multible line comments or syntactical units that can be handled like line comments (
addLineComment methods),
- support for multible block comments or syntactical units that can be handled like block comments (
addBlockComment methods),
- support for multible strings or similar syntactical units, for instance characters (
addString methods),
- support for special sequences (
addSpecialSequence methods,
- the usual simple whitespaces and separators (
setWhitespaces and setSeparators),
- support for keywords (
addKeyword methods),
More examples can be found in the various JUnit tests for the Tokenizer.
The basic behaviour of the tokenizer can be configured with various flags, that may control the parsing process as a whole, but also the handling of single comments, keywords or special sequences. This is shown in the example above: HTML is generally not case-sensitive, but some special characters are, like ä (ä) and Ä (Ä). Currently, the tokenizer recognizes the following parse flags:
F_NO_CASE: Comparison is done ignoring the uppercase / lowercase differences between letters. 'A' is equal 'a'. Programming languages like Pasqal, ORACLE's PL/SQL and Ada fall in that category. On the other hand, Java, C and C++ are case-sensitive.F_KEYWORDS_NO_CASE and F_KEYWORDS_CASE: If the general compare operations are done case-sensitive, but keywords are not case-sensitive, or vice versa, these flags cover such situations.F_RETURN_WHITESPACES:In many cases, parsers are not interested in whitespaces. If in certain circumstances, whitespaces and comments should be returned as tokens to the caller, use this flag.
F_TOKEN_POS_ONLY: For perfomance and memory reasons, this flag is used to avoid copying found token images (the string representing the token) to the Token variable, returned by the parse methods nextToken, next and others.F_KEEP_DATA: Set this flag to let the tokenizer buffer all data. Normally, a tokenizer keeps only the amount of data that fits in an internal buffer. This flag ensures, that the internal buffer is dynamically enlarged to store all data.F_COUNT_LINES: Tells the tokenizer to count lines and columns. That is not really a performance boost ;-).F_ALLOW_NESTED_COMMENTS: Nested block comments are usually not allowed. This flag changes the default behaviour.F_PARSE_NUMBERS: With this flag, the tokenizer tries to identify numbers (not yet implemented).
The mentioned flags can also be set for a single comment, keyword or special sequence. Currently, only the F_NO_CASE (or its absence) takes effect. But this is the most important.
The more advanced add... methods have a parameter, called the companion. This is any additional information, the user wants to be associated with a certain keyword, special sequence, comment or string. In the example above, the HTML codes for the german Umlaute ä etc. have associated the actual german letter to them.
If a specific flag can be set or cleared while the parsing process is under way, depends highly on the implementation. Our basic version supports dynamic change for the following flags:
F_RETURN_WHITESPACESF_TOKEN_POS_ONLYF_COUNT_LINES (clearing the flag is save, setting it works correctly except that line 0, column 0 is the position, where the flag was set).
The results of the Tokenizers parse operations a delivered as objects of class src/de/susebox/util/Token.java. Depending on some of the control flags set for the Tokenizer, some or all properties of the Token object are filled. Without the flag F_COUNT_LINES the result values of the methods getStartLine, getStartColumn etc. are useless. The flag F_TOKEN_POS_ONLY lets getToken return null.
Java classes and documentation:
Class / Interface |
Doc. |
Remarks |
Tokenizer
|
here |
The interface with parsing operations, getter and setter methods for properties and raw data access |
AbstractTokenizer
|
here |
Implementation of the Tokenizerinterface. Extensions need to implement an InputStream-like read method. |
InputStreamTokenizer
|
here |
Derived from AbstractTokenizer. Reads data from an InputStream. |
Token
|
here |
Token description (image, type, positions and length). Objects of this class are returned by the parse methods of Tokenizer. |
TokenizerProperty
|
here |
Description of comments, keywords, special sequences and strings. Objects of this class are returned by the various Enumeration methods of Tokenizer. |
TokenizerException
|
here |
Common exception thrown by the methods of Tokenizer. Supports the ExceptionList interface |
|
|
Planned modules and expected improvements:
We are planning to supply class(es) for more sophisticated parse operations like finding the closing sequence for a given opening one (for instance, the matching '}' for a found '{' in Java), parsing various number and date formats etc.
Currently, there are only the basic strategies to improve speed implemented (Use of ordered arrays for binary search, hash tables). Also, there is not much consideration regarding memory usage.
But most important is feedback :-)
|
|
Beside the easy-to-use approach of the generic tokenizer described above, there is also the so-called plugin tokenizer. It is an implementation of the Tokenizer.java interface, derived from the class AbstractTokenizer.java. It can be used as an alternative to the InputStreamTokenizer.java.
The plugin tokenizer may be faster in various situations. This might especially be true for parsing XML or HTML tags and treating the real contents as block comments. But don't expect double speed, the generic tokenizer is not that slow ;-)
More important, it provides some means to deal with situations where the public methods of the generic tokenizer are not good / nice / convenient enough.
Here is an example:
// Imports
import java.io.FileInputStream;
import java.io.InputStreamReader;
import java.util.ArrayList;
import de.susebox.java.util.Token;
import de.susebox.java.util.Tokenizer;
import de.susebox.java.util.TokenizerProperty;
import de.susebox.jtopas.PluginTokenizer;
import de.susebox.jtopas.InputStreamSource;
import de.susebox.jtopas.SequenceHandler;
import de.susebox.jtopas.SeparatorHandler;
// class to hold main method
public class XMLTokenizer implements SequenceHandler, SeparatorHandler {
// class constants
private static final Object START_TAG_COMP = new Object;
private static final Object END_TAG_COMP = new Object;
private static final TokenizerProperty STRING_PROP
= new TokenizerProperty(Token.STRING, new String[] { "\"", "\"", "\\" }, null );
private static final TokenizerProperty COMMENT_PROP
= new TokenizerProperty(Token.BLOCK_COMMENT, new String[] { "<!--", "-->" }, null );
private static final TokenizerProperty SPEC_COMMENT_PROP
= new TokenizerProperty(Token.SPECIAL_SEQUENCE, new String[] { "<!" }, null );
private static final TokenizerProperty END_TAG
= new TokenizerProperty(Token.SPECIAL_SEQUENCE, new String[] { "</" }, END_TAG_COMP );
private static final TokenizerProperty START_TAG
= new TokenizerProperty(Token.SPECIAL_SEQUENCE, new String[] { "<" }, START_TAG_COMP );
private static final TokenizerProperty TAG_END
= new TokenizerProperty(Token.SPECIAL_SEQUENCE, new String[] { ">" }, null );
// member variables
private PluginTokenizer _myTokenizer = null;
private ArrayList _tagList = new ArrayList(1024);;
// Main method. Supply a XML file name as argument
public static void main(String[] args) {
FileInputStream stream = new FileInputStream(args[0]);
InputStreamReader reader = new InputStreamReader(stream);
PluginTokenizer tokenizer = new PluginTokenizer();
// setup the tokenizer
tokenizer.setSource(new InputStreamSource(reader));
tokenizer.setParseFlags( Tokenizer.F_TOKEN_POS_ONLY
| Tokenizer.F_RETURN_WHITESPACES);
tokenizer.setSequenceHandler(this);
tokenizer.setSeparatorHandler(this);
// get all the XML tags
while (tokenizer.hasMoreToken()) {
Token token = tokenizer.nextToken();
if (token.getType() == Token.SPECIAL_SEQUENCE) {
if (token.getCompanion() == START_TAG_COMP) {
token = tokenizer.nextToken();
_tagList.add(new TagInfo(tokenizer.current(), START_TAG_COMP));
} else if (token.getCompanion() == END_TAG_COMP) {
token = tokenizer.nextToken();
_tagList.add(new TagInfo(tokenizer.current(), END_TAG_COMP));
}
}
}
}
// Plugin interface method
public void setTokenizer(PluginTokenizer tokenizer) {
_myTokenizer = tokenizer;
}
// SeparatorHandler interface method
public boolean isSeparator(char testChar) {
return testChar == '=';
}
// get the maximum possible length for a special sequence
public int getSequenceMaxLength() {
return 4; // length of "<!--"
}
// The sequence handler
public TokenizerProperty isSequenceCommentOrString(int startAt, int maxChars) {
TokenizerProperty prop = null;
switch (_myTokenizer.getCharUnchecked(startingAtPos)) {
case '"':
// strings are attribute values in XML
prop = STRING_PROP;
break;
case '<':
// tag opening
switch (_tokenizer.getCharUnchecked(startingAtPos + 1)) {
case '!':
if ( _myTokenizer.getCharUnchecked(startingAtPos + 2) == '-'
&& _myTokenizer.getCharUnchecked(startingAtPos + 3) == '-') {
prop = COMMENT_PROP;
} else {
prop = SPEC_COMMENT_PROP;
}
break;
case '/':
prop = END_TAG;
break;
default:
prop = START_TAG;
break;
}
break;
case '>':
// tag closing
prop = TAG_END;
break;
}
}
}
Beside the handler methods, there is one big difference to the generic tokenizer. No comments, special sequences and separators are added to the plugin tokenizer. While it would have been possible to do so, the tokenizer wouldn't have heeded these properties, since the decision about what is a special sequence and what not, is made by the handler isSequenceCommentOrString.
Java classes and documentation:
Class / Interface |
Doc. |
Remarks |
PluginTokenizer
|
here |
This class extends the AbstractTokenizer.java. It is possible to "plug in" various handlers like a whitespace or separator handler. |
Plugin
|
here |
Root interface for all plug-ins. |
TokenizerSource
|
here |
Interface for a data source plugin. Such plugins provide data to the tokenizer from whatever source is desired. |
InputStreamSource
|
here |
Simple implementation of the TokenizerSource interface, based on a java.io.Reader. |
SeparatorHandler
|
here |
Interface for handling simple separators. Separators are single characters like brackerts, colons, slashes or whatever has a similar meaning. |
SequenceHandler
|
here |
Interface for processing special sequences like the !=, && or += operators of C and Java. |
WhitespaceHandler
|
here |
Interface for processing whitespaces excluding line and block comments. |
|
|
Planned modules and expected improvements:
Currently, the PluginTokenizer provides only some of the most obvious plug-in points. More read handlers like the readWhitespaces could be usefull, especially for comments and strings.
|
|
In order to see, whats going wrong in one's Java application, ex.printStackTrace() is often perfectly sufficient. As long as You feel comfortable with this, don't bother about exception stacks ;-)
But the following problems occur while working with exceptions:
- A method throws an exception, for instance containing the message "file not found". The calling method catches this exception to add its own exception, for instance "could not read properties". With standard Java exceptions, there is no elegant way to keep the first exception while throwing the second one.
- An interface requires that only
IOException are thrown by a specific method. One's implementation, however, may throw SQLException by calling some JDBC methods. Usually, one would take the message of the SQLException only to put it into a newly created IOException. This is not really a nice solution.
- There are more complex exception situations, for instance a set of parameters where one may conflict with another or some of them may be invalid. A good example are
MessagingException's in the Java Mail API, there quite a lot of things may go wrong (no mail server, no network ...) because of a lot of other things (invalid mail addresses, unknown attachments ...). How can one bundle a variety of problems into one exception to be thrown?
We use the interface ExceptionList.java to deal with these situations. It is a lightweight interface with only two methods:
nextException: This method returns the succeeding exception in an exception list. Seeing the exception list as a stack, the next exception is the one under the current in the stack order.isWrapperException: Returns true, if the current exception only contains an inner exception. This is the usually the case in the situation explained in the second case above.
We use either exceptions derived from the standard JDK exceptions or our own ones. Both support the ExceptionList interface. Unfortunately, at least for the exception list aspect, Java doesn't support multiple inheritance. The implementation code for the ExceptionList interface must be written down for each derived exception class. We provide the expected implementation code in the interface (in block comments) to make coding easier.
Exception messages can be provided in the usual way by composing the message when constructing the exception. Another way is the usage of a format string and a array of parameters. The latter approach avoids the costly formatting when an exception is thrown (nobody knows, if the message is actually shown somewhere). Another aspect is the separation of language-depending format and the actual parameters.
Beside the methods of the ExceptionList interface, all our exceptions implement the getMessage method of the java.lang.Throwable class. The first reason is to provide a message depending on the fact if an exception is a wrapper exception or not. The second reason is the use of the java.text.MessageFormat class to assemble messages from format strings and a parameter array.
Java classes and documentation:
Class / Interface |
Doc. |
Remarks |
ExceptionList
|
here |
The interface for nested and wrapped exceptions |
ExtRuntimeException
|
here |
Implementation of the ExceptionList interface for the JDK exception java.lang.RuntimeException |
ExtIndexOutOfBoundsException
|
| here |
Implementation of the ExceptionList interface for the JDK exception java.lang.IndexOutOfBoundsException |
ExtIOException
|
here |
Implementation of the ExceptionList interface for the JDK exception java.io.IOException |
|
|
Planned modules and expected improvements:
There are a lot of JDK exceptions that haven't been derived so far. A growing list of such derivations should be included in new versions of JTopas.
|
|
To compile, use and run sources and / or archives of JTopas, You need at least a Java runtime environment (JRE), that can be obtained - among others - from Sun. For debugging purposes or a look inside the JDK sources, the full JDK is a better choice. For Linux users: there should be at least one JDK in Your distribution.
Since Version 0.3 of JTopas, a Java 2 environment (at least JDK 1.2) is nessecary. Classes like java.util.Vector, java.util.Hashtable and especially java.util.Enumeration are abandoned in favour of java.util.ArrayList, java.util.HashMap and java.util.Iterator.
For development, we use the Netbeans IDE with JDK 1.3.1 on a SuSE Linux 7.3. The Jakarta Ant build tool is used to compile, archive, javadoc and package the whole thing. CVS does the version control. There are some ant targets for CVS.
|