JTopas - Java tokenizer and parser tools

Version 0.6

Contents:

1. What is this?
2. The generic tokenizer
3. Working with embedded tokenizers
4. The service provider interface
5. Using exception stacks
6. Using the Environment interface
7. Directories and Java package structure
8. Configuring JUnit tests
9. Environment
10. Installation
11. Building with Ant

What is this?

As the title indicates, this is software written in Java and designed for being used in Java applications, libraries and other Java environments. In this version, the following features are provided:

• a generic, multi-purpose tokenizer for "readable" text (source code, HTML, XML, ASCII text ...) featuring line and block comment, string, keyword, special sequence and pattern support,
• an interface for exception stacks (nested exceptions, as already known from java.sql.SQLException and javax.mail.MessagingException).
• a substitute for the standard IO features of the JDK class java.lang.System.

This list will be frequently extended by new features, implementations of standard Java API's, and, last but not least, extensions to existing features.

Currently, JTopas consists of:

• the Java archive jtopas-core.jar with the most recent interfaces and classes for Java programs that like to use JTopas,
• the Java archive jtopas-compat.jar with the deprecated interfaces and classes for backward compatibility,
• the Java archive jtopas.jar that combines the contents of jtopas-core.jar and jtopas-compat.jar for users who don't want to or cannot change existing class path settings,
• the Java archive jtopas-jt.jar containing some JUnit tests for those who like to check the functionality of the jtopas.jar,
• the documentation of the interfaces and classes in the archives jtopas-core.jar and jtopas-compat.jar, generated with the javadoc tool of the Java Development Kit (JDK),
• the contents of our project web site,
• a build file build.xml for the ant utility of the Apache Jakarta Project and
• various license files (COPYING, GNU-GPL.txt, GNU-LGPL.txt), a RELEASE-NOTES file, a changelog (CHANGES) and this README.

NOTE: The java package root is "de.susebox" according to our registered internet domain, not jtopas or similar.

The generic tokenizer

If You need to parse more sophisticated texts than can easily be handled using java.util.StringTokenizer or java.io.StreamTokenizer, and You dont want to use JavaCC or JTB or other parser generators, our generic, multi-purpose tokenizer may be just right. It has more abilities to tokenize source code, HTML etc. than the mentioned JDK classes without the complexity and power of parser generators. In fact, with a more or less simple wrapper it can be used as a tokenizer for JavaCC and JTB, if the generated tokenizers do not fit for some reason.

In the following example, we extract the contents from a HTML file to produce a roughly formatted text file. A more sophisticated version can be found in one of the JUnit tests (see junit/de/susebox/jtopas/TestStandardTokenizer.java, Method testContentsFormatting):

// Imports
import java.io.FileInputStream;
import java.io.InputStreamReader;

import de.susebox.jtopas.Token;
import de.susebox.jtopas.Tokenizer;
import de.susebox.jtopas.TokenizerProperties;
import de.susebox.jtopas.StandardTokenizer;
import de.susebox.jtopas.StandardTokenizerProperties;
import de.susebox.jtopas.ReaderSource;

// class to hold main method
public class ContentsExtractor {

  // Main method. Supply a HTML file name as argument
  public static void main(String[] args) {
    FileInputStream     stream    = new FileInputStream(args[0]);
    InputStreamReader   reader    = new InputStreamReader(stream);
    TokenizerProperties props     = new StandardTokenizerProperties();
    Tokenizer           tokenizer = new StandardTokenizer();
    Token               token;
    int                 len;
    int                 caseFlags; 

    // setup the tokenizer
    props.setParseFlags( TokenizerProperties.F_NO_CASE 
                       | TokenizerProperties.F_TOKEN_POS_ONLY 
                       | TokenizerProperties.F_RETURN_WHITESPACES);
    caseFlags = props.getParseFlags() & ~Tokenizer.F_NO_CASE;
    props.setSeparators(null);
    props.addBlockComment("<", ">");
    props.addBlockComment("<HEAD>", "</HEAD>");
    props.addBlockComment("<!--", "-->");
    props.addSpecialSequence("&lt;", "<");
    props.addSpecialSequence("&gt;", ">");                   
    props.addSpecialSequence("&auml;", "ä", caseFlags);
    props.addSpecialSequence("&Auml;", "Ä", caseFlags);
    props.addSpecialSequence("&ouml;", "ö", caseFlags);
    props.addSpecialSequence("&Ouml;", "Ö", caseFlags);
    props.addSpecialSequence("&uuml;", "ü", caseFlags);
    props.addSpecialSequence("&Uuml;", "Ü", caseFlags);
    props.addSpecialSequence("<b>", "");
    props.addSpecialSequence("</b>", "");
    props.addSpecialSequence("<i>", "");
    props.addSpecialSequence("</i>", "");
    props.addSpecialSequence("<code>", "");
    props.addSpecialSequence("</code>", "");

    tokenizer.setTokenizerProperties(props);
    tokenizer.setSource(new ReaderSource(reader));

    // tokenize the file and print basically
    // formatted context to stdout
    len = 0;
    while (tokenizer.hasMoreToken()) {
      token = tokenizer.nextToken();
      switch (token.getType()) {
      case Token.NORMAL:
        System.out.print(tokenizer.currentImage());
        len += token.getLength();
        break;
      case Token.SPECIAL_SEQUENCE:
        System.out.print((String)token.getCompanion());
        break;
      case Token.BLOCK_COMMENT:
        if (len > 0) {
          System.out.println();
          len = 0;
        }
        break;
      case Token.WHITESPACE:
        if (len > 75) {
          System.out.println();
          len = 0;
        } else if (len > 0) {
          System.out.print(' ');
          len++;
        }
        break;
      }
    }
  }
}
	    

There are two core interfaces involved in the above example: TokenizerProperties declares methods to configure the tokenizer while Tokenizer is for the actual tokenizing process. These interfaces are implemented in the classes StandardTokenizerProperties and StandardTokenizer that should suffice for a wide range of programming tasks.

The generic tokenizer supports the following features, some of them can be found in the example above:

• support for multible line comments or syntactical units that can be handled like line comments (addLineComment methods),
• support for multible block comments or syntactical units that can be handled like block comments (addBlockComment methods),
• support for multible strings or similar syntactical units, for instance characters (addString methods),
• support for special sequences (addSpecialSequence methods,
• the usual simple whitespaces and separators (setWhitespaces and setSeparators),
• support for keywords (addKeyword methods),
• support for regular expressions (addPattern methods), if a backing regex library is present (like the java.util.regex package in JDK 1.4).

More examples can be found in the various JUnit tests for the Tokenizer.

The basic behaviour of the tokenizer can be configured with various flags, that may control the parsing process as a whole, but also the handling of single comments, keywords or special sequences. This is shown in the example above: HTML is generally not case-sensitive, but some special characters are, like ä (ä) and Ä (Ä). Currently, the tokenizer recognizes the following parse flags:

• F_CASE: Comparison is done exactly. 'A' does not equals 'a'. Programming languages like C, Java and the UNIX shells fall in that category.
• F_NO_CASE: Comparison is done ignoring the uppercase / lowercase differences between letters. 'A' is equal 'a'. Programming languages like Pasqal, ORACLE's PL/SQL and Ada fall in that category. On the other hand, Java, C and C++ are case-sensitive.
• F_RETURN_WHITESPACES: In many cases, parsers are not interested in whitespaces. If in certain circumstances, whitespaces and comments should be returned as tokens to the caller, use this flag.
• F_TOKEN_POS_ONLY: For perfomance and memory reasons, this flag is used to avoid copying found token images (the string representing the token) to the Token variable, returned by the parse methods nextToken, nextImage and others.
• F_KEEP_DATA: Set this flag to let the tokenizer buffer all data. Normally, a tokenizer keeps only the amount of data that fits in an internal buffer. This flag ensures, that the internal buffer is dynamically enlarged to store all data.
• F_COUNT_LINES: Tells the tokenizer to count lines and columns. That is not really a performance boost ;-).
• F_ALLOW_NESTED_COMMENTS: Nested block comments are usually not allowed. This flag changes the default behaviour.

The mentioned flags can also be set for a single comment, keyword or special sequence. Currently, only the F_NO_CASE (or its absence) takes effect. But this is the most important.

The more advanced add... methods have a parameter, called the companion. This is any additional information, the user wants to be associated with a certain keyword, special sequence, comment or string. In the example above, the HTML codes for the german Umlaute ä etc. have associated the actual german letter to them.

If a specific flag can be set or cleared while the parsing process is under way, depends highly on the implementation. Our basic version supports dynamic change for the following flags:

• F_RETURN_WHITESPACES
• F_TOKEN_POS_ONLY
• F_COUNT_LINES (clearing the flag is save, setting it works correctly except that line 0, column 0 is the position, where the flag was set).

The results of the Tokenizers parse operations a delivered as objects of class Token.java. Depending on some of the control flags set for the Tokenizer, some or all properties of the Token object are filled. Without the flag F_COUNT_LINES the result values of the methods getStartLine, getStartColumn etc. are useless. The flag F_TOKEN_POS_ONLY lets getImage return null.

With version 0.6, pattern handling is added to the basic tokenizer operations. Using this new feature requires a backing library for regular expression support. Since JDK 1.4 the Java standard edition contains the package java.util.regex that is used by StandardTokenizerProperties to resolve pattern. Pattern can be used to extract number and date token or even whole expressions. Please keep in mind that pattern matching is a more complex operation that can slow down the tokenizing process when used extensively.

Java classes and documentation:

Working with embedded tokenizers

Beside the implementation of the Tokenizer methods the class StandardTokenizer provides support for embedded tokenizers. This feature is useful in situations where one input source contains parts with different sets of tokenizer properties. This is for instance the case in common HTML files where HTML is mixed with CSS (Cascading Style Sheets) and JavaScript. Another example are the javadoc comments inside Java sources.

Two examples for embedded tokenizers can be found in one of the JUnit test cases, TestEmbeddedTokenizer.

The idea behind the embedded tokenizers is that one master tokenizer contains the input buffer and controls the read operation while the slave tokenizers share the input buffer. The user switches actively between the tokenizers when boundaries between different parts are reached (end of block comment in javadoc comments, for instance). The following code snippet shows the principle:

StandardTokenizerProperties prop1    = new StandardTokenizerProperties();
StandardTokenizerProperties prop2    = new StandardTokenizerProperties();
StandardTokenizer           base     = new StandardTokenizer(prop1);
StandardTokenizer           embedded = new StandardTokenizer(prop2);

// setting properties (comments, keywords etc.)
prop1.setSpecialSequence("/**", embedded);
prop2.setSpecialSequence("*/",  base);

// embedding a tokenizer
base.addTokenizer(embedded);

// tokenizing with base
StandardTokenizer current = base;

while (current.hasMoreToken()) {
  Token token = current.nextToken();

  if (   token.getType == Token.SPECIAL_SEQUENCE 
      && token.getCompanion() instanceof StandardTokenizer) {
    current.switchTo((StandardTokenizer)token.getCompanion());
    current = token.getCompanion();
  }
}
	    

The Service Provider Interface

In former versions of JTopas, there was the so-called plugin tokenizer. The concept a bit complicated and only an add-on to the (old version) of the core tokenizer classes. With version 0.6, the old design was completely renovated and the plugin tokenizer has been replaced by a Service Provider Interface (SPI).

The SPI is used by the StandardTokenizer to communicate with the associated TokenizerProperties object. The tokenizer is therefore independend from the actual implementation of the TokenizerProperties interface. Moreover, the functionality of this interface can be partially or even completely shifted to peer classes implementing various interfaces in the de.susebox.jtopas.spi package.

The SPI can basically be used in the following ways:

• Modules that already have their own setup catalogues / search structures for keywords, operators etc., can implement the approbriate interface as a bridge between a Tokenizer and their own classes.
• In situations where the type of a token depends on internal states or additional information that cannot be handled by the StandardTokenizerProperties.
• When implementing Your own TokenizerProperties class it is often a good choice to implement various SPI interfaces along with it. See for instance the StandardTokenizerProperties class.

Here is an example there a lookup structure for a set of keywords already exists. The situation is a bit artificial, but its the principle that counts :-)

// Imports
import java.util.Collection;
import java.io.FileInputStream;
import java.io.InputStreamReader;

import de.susebox.jtopas.Token;
import de.susebox.jtopas.TokenizerProperty;
import de.susebox.jtopas.ReaderSource;
import de.susebox.jtopas.spi.KeywordHandler;
import de.susebox.jtopas.spi.DataProvider;


// class to hold main method
public class MyKeywordHandler 
  implements de.susebox.jtopas.spi.KeywordHandler {

  /** the keyword collection */
  private Collection _keywords = null;

  /** Constructor taking the existing keyword collection */
  public MyKeywordHandler(Collection keywords) {
    _keywords = keywords;
  }

  /** Method from the KeywordHandler interface */
  public TokenizerProperty isKeyword(DataProvider dataProvider) {
    if (_keywords != null) {
      String keyword = new String(dataProvider.getData(), 
                                    dataProvider.getStartPosition(),
                                    dataProvider.getLength());
      
      if(_keywords.contains(keyword)) {
        return new TokenizerProperty(Token.KEYWORD, 
                                     new String[] { keyword }, 
                                     null, 
                                     TokenizerProperties.F_CASE); 
      } else {
        return null;
      }
    } else {
      return false;
    }
  }
}

/**
* Class searching for keywords in an input stream. The files
* may contain line comments starting with the '#' character
* and strings encapsulated in '"' or ''' pairs
*/
public class KeywordFinder {

  /** 
  * Main method taking file names as data sources.
  * With the '-k' option keywords can be specified, that should 
  * be found in the files.
  */
  public void main(String[] args) {

    // Setting up the keyword structure
    HashSet keywords  = new HashSet();

    for (int index = 0; index < args.length; ++index) {
      String arg = args[index];

      if (arg.length() > 2 && arg.charAt(0) == '-' && arg.charAt(1) == 'k') {
        keywords.add(arg.substring(2);
      }
    }

    // Setting up the tokenizer
    StandardTokenizerProperties props     = new StandardTokenizerProperties();
    StandardTokenizer           tokenizer = new StandardTokenizer(props);
    MyKeywordHandler            handler   = new MyKeywordHandler(keywords);

    props.setParseFlags(TokenizerProperties.F_COUNT_LINES);
    props.addLineComment("#");
    props.addString("\"", "\"", "\\");
    props.addString("'", "'", "\\");

    tokenizer.setKeywordHandler(handler);

    // tokenizing the given files
    for (int index = 0; index < args.length; ++index) {
      String arg = args[index];

      if (arg.length() > 1 && arg.charAt(0) != '-') {
        FileInputStream is = new FileInputStream(arg);

        tokenizer.setSource(new ReaderSource(new InputStreamReader(is)));

        while (tokenizer.hasMoreToken()) {
          Token token = tokenizer.nextToken();

          switch (token.getType()) {
          case Token.KEYWORD:
            System.out.println(arg 
                             + ": at " + token.getStartLine() 
                             + "/" + token.getStartColumn()
                             + ": " + token.getImage());
            break;
          }
        }
        is.close();
      }
    }
  }
}

	    

The example could be altered, so that the keyword handler class reads keywords from a file that changes frequently during runtime of the tokenizer. Ot where the currently active set of keywords is read from a database. Equivalent to the shown keyword handling, there are interfaces to control separator, whitespace, comment, pattern and special sequence detection separately.

Java classes and documentation:

Using exception stacks

In order to see, whats going wrong in one's Java application, Throwable.printStackTrace() is often perfectly sufficient. As long as You feel comfortable with this, don't bother about exception stacks ;-)

But the following problems occur while working with exceptions (Objects of type java.lang.Throwable, subclasses are java.lang.Exception and java.lang.Error):

• A method throws an exception, for instance containing the message "file not found". The calling method catches this exception to add its own exception, for instance "could not read properties". With standard Java exceptions prior to JDK 1.4, there was no elegant way to keep the first exception while throwing the second one.
• An interface requires that only IOException are thrown by a specific method. One's implementation, however, may throw SQLException by calling some JDBC methods. Usually, one would take the message of the SQLException only to put it into a newly created IOException. This is not really a nice solution.
• There are more complex exception situations, for instance a set of parameters where one may conflict with another or some of them may be invalid. A good example are MessagingException's in the Java Mail API, there quite a lot of things may go wrong (no mail server, no network ...) because of a lot of other things (invalid mail addresses, unknown attachments ...). How can one bundle a variety of problems into one exception to be thrown?

We use the interface ThrowableList.java to deal with these situations. It is a lightweight interface with only two methods:

• nextThrowable: This method returns the succeeding exception in an exception list. Seeing the exception list as a stack, the next exception is the one under the current in the stack order.
• isWrapper: Returns true, if the current exception only contains an inner exception. This is the usually the case in the situation explained in the second szenario above.

We use either exceptions derived from the standard JDK exceptions or our own ones. Both support the ThrowableList interface. Unfortunately, at least for the exception list aspect, Java doesn't support multiple inheritance. The implementation code for the ThrowableList interface must be written down for each derived exception class. We provide the expected implementation code in the interface (in block comments) to make coding easier.

Exception messages can be provided in the usual way by composing the message when constructing the exception. Another way is the usage of a format string and a array of parameters. The latter approach avoids the costly formatting when an exception is thrown (nobody knows, if the message is actually shown somewhere). Another aspect is the separation of language-depending format and the actual parameters.

Beside the methods of the ThrowableList interface, all our exceptions implement the getMessage method of the java.lang.Throwable class. The first reason is to provide a message depending on the fact if an exception is a wrapper exception or not. The second reason is the use of the java.text.MessageFormat class to assemble messages from format strings and a parameter array.

Future releases of JTopas will reflect the new changed exception facility of JDK 1.4. In particular, nextThrowable method implementations should call Throwable.getCause(). The method itself should be deprecated in favor of Throwable.getCause() as soon as JDK versions prior to JDKL 1.4 are sufficiently out-of-date.

Java classes and documentation:

Using the Environment interface

Sometimes standalone classes that serve a simple purpose should be integrated in a more complex software. Unfortunately, they contain calls to System.out or even System.exit. By using the de.susebox.java.lang.Environment it is very easy to connect such classes to a more sophisticated input/output system or catch the exit call without actually switching of the Java Virtual Machine :-)

Moreover, the classes in question continue to function standalone as before. How this is done should be shown in the following example. First the class before Environment is used:

/** 
* Simple Hello something class containing only a main method.
* All arguments are written to the standard output channel.
*/
public class Echo {
 
  public static void main(String[] argv) {
    // check the parameter
    int exitCode = 0;

    if (argv == null) {
      System.err.println("Nothing to echo :-(");
      exitCode = 1;
    } else
      // output all elements of the parameter
      for (int index = 0; index < argv.length; ++index) {
        System.out.println(argv[index]);
      }
      exitCode = 0;
    }
    System.exit(exitCode);
  }
}
	      

Using the Environment interface leads to the following, slightly changed class, that has the identical standalone behaviour as the one shown above:

// Imports
import de.susebox.java.lang.Environment;
import de.susebox.java.lang.EnvironmentProvider;


/** 
* Simple Hello something class containing only a main method.
* All arguments are written to the standard output channel.
*/
public class Echo {
 
  public static void main(String[] argv) {
    Environment env = EnvironmentProvider.getEnvironment(Echo.class);

    if (argv == null) {
      env.err().println("Nothing to echo :-(");
      env.setExitStatus(1);
    } else {
      // output all elements of the parameter
      for (int index = 0; index  < argv.length; ++index) {
        env.out().println(argv[index]);
      }
      env.setExitStatus(0);
    }
    env.exit();
  }
}
	    

This second version of the Echo class can now be used inside another class that likes to use the services of Echo. But instead of printing the arguments to standard output System.out, the output should be collected in a java.lang.String:

// Imports
import java.io.ByteArrayOutputStream;
import java.io.PrintStream;

import de.susebox.java.lang.Environment;
import de.susebox.java.lang.EnvironmentProvider;

public class StringUtilities {

  /** 
  * Environment implementation
  */
  class StringWriterEnvironment implements Environment {

    private PrintStream _out        = System.out;
    private int         _exitStatus = 0;

    public StringWriterEnvironment(PrintStream ps) {
      if (ps != null) {
        _out = ps;
      }
    }

    public InputStream in()  { return System.in; }
    public PrintStream out() { return _out;      }
    public PrintStream err() { return out();     }

    public void setExitStatus(int status) { _exitStatus = status; }
    public int  getExitStatus()           { return _exitStatus;   }
    public void exit() {}
  }
 
  /** 
  * Getting all arguments of a string array into one string
  */
  public String array2String(String[] argv) {
    ByteArrayOutputStream   os  = new ByteArrayOutputStream();
    PrintStream             ps  = new PrintStream(os);
    StringWriterEnvironment env = new StringWriterEnvironment(ps);

    EnvironmentProvider.setEnvironment(Echo.class, env);
    Echo.main(argv);
    if (env.getExitStatus() != 0) {
      throw new RuntimeException(os.toString()); 
    }
    return os.toString();
  }
}
	    

While it possible to redirect the standard input, output and error channels in java.lang.System, that way changes the channels for all currently active objects. Using Environment and EnvironmentProvider it is possible to assign different channels on a per class base (as in the example above) and even for single objects.

Java classes and documentation:

Directories and Java package structure

After extracting files from the JTopas tar'ed and gzip'ed archive jtopas-0.1.tar.gz, the following directories can be found:

The Java package structure follows the usual policy. The root for our package hierarchy is derived from our web domain: de.susebox. For application-independed interfaces and classes that are extended from or related to JDK (J2SE, J2EE) API's, we use the equivalent structure. If, for instance, we would implement a java.io.InputStream, it would appear in the package de.susebox.java.io.

Configuring JUnit tests

We use the popular JUnit framework for testing our classes. The tests are composed hierarchically by the (static) suite method of JUnit. The test class de.susebox.java.JavaTestSuite assembles a suite of all subpackage test suites. Part of it is, for instance, de.susebox.java.util.UtilTestSuite. The root test class is de.susebox.SuseboxTestSuite.

While some test cases are typical unit tests as JUnit suggests, there are others, that can - and need to - be configured using a configuration file. Currently, these configuration files contain entries that can be read by the java.lang.Class.getResourceAsStream method. To run the tests successfully, You should:

1. edit the configuration files to contain valid paths, data etc. in Your specific environment and
2. change into the $JTOPAS_HOME/junit directory and add "." (current working directory) to the classpath. That way, the class loader will eventually find the configuration files.

It is also possible to put all *.conf files into a separate directory tree, provided they have the same relative paths as in the $JTOPAS_HOME/junit subtree

The configuration files contain several test data sets. Each set is a collection of properties defined in the test case class using the configuration file, followed by the number of the test data set:

  # first test set
  Path1=README.html

  # second test set
  Path2=build.xml
	      

Line comments may be used as shown above. The property base names ("Path" in the example) are defined in the test case Java classes. You will find the declaration at the beginning of the class sources. See for instance TestInputStreamTokenizer.java.

Environment

To compile, use and run sources and / or archives of JTopas, You need at least a Java runtime environment (JRE), that can be obtained - among others - from Sun. For debugging purposes or a look inside the JDK sources, the full JDK is a better choice. For Linux users: there should be at least one JDK in Your distribution.

Since Version 0.3 of JTopas, a Java 2 environment (at least JDK 1.2) is nessecary. The older java.util classes like Vector, Hashtable and especially Enumeration are abandoned in favour of ArrayList, HashMap and Iterator.

For development, we use the Netbeans IDE with Sun's JDK 1.4.1 on a SuSE Linux 8.1. The Jakarta Ant build tool is used to compile, archive, javadoc and package the whole thing. CVS does the version control. There are some ant targets for CVS.

Installation

Our development platform is Linux, therefore we provide the usual zipped tar archive jtopas-version.tar.gz. The version tag has the form major.minor[.maintenance], for instance 0.1 or 149.73.3

Installation is straightforward:

me@ours: > cd <parent_dir>
me@ours: > tar xvzf <download_dir/>jtopas-0.6.tar.gz

For the various Microsoft operating systems, programs like Winzip or Power Archiver are able to extract the tgz-archive. With the Cygwin bash shell for Windows, one may install the archive like in Unix environment.

JTopas requires JDK 1.2 or higher to run (mainly, we use the collection classes not available in JDK 1.1). Recommended and - for the use of pattern matching - is JDK 1.4. For the JUnit tests, JUnit 3.7 or higher is nessecary. All archives have been tested against Sun's JDK 1.3.1 and JDK 1.4.1 on a SuSE Linux 8.1. Currently, IBM's JDK 1.3.0 refuses to run on my Linux, but there shouldn't be a problem building JTopas with it ;-)

Building with Ant

The Jakarta Ant tool uses a build file, written in an XML dialect, as an equivalent to the classical Makefile.

The default name is build.xml and such a file exists for JTopas (look here).