Release notes for JTopas:
-------------------------

Version 0.8
-----------
This is a major release with a variety of new features, that help to use JTopas
in other contexts beside the stream-oriented way that was the one and only way
for JTopas.

Deprecated classes removed:
The old version of the tokenizer classes (mainly de.susebox.java.util.* and the
plugin interfaces in de.susebox.jtopas) that has been deprecated since 
JTopas 0.6, has been removed.

Interface de.susebox.jtopas.Flags:
This new interface contains the control flags (F_... constants) previously 
defined in TokenizerProperties. It reduces cross-references and is shorter to
write :-). To keep existing code running, the interface is implemented by 
TokenizerProperties. However, it is suggested to globally replace all 
"TokenizerProperties.F_" occurences against "Flags.F_" in all Your source code.

TokenizerProperties interface:
The interface contains the new methods addWhitespaces, removeWhitespaces, 
addSeparators and removeSeparators for easier handling of these token classes.
For instance, it is much easier to remove the character '.' from the default 
separator set than constructing a separator set without the '.' and setting it
with TokenizerProperties.setSeparators.

Tokenizer and TokenizerProperties implementations:
The core functionality of these interfaces is now bundled in two abstract classes,
AbstractTokenizer and AbstractTokenizerProperties. The AbstractTokenizer class
separates the actual token detection from the data supply, thus allowing for other
data origins than the java.io-based approach of the previous JTopas versions.

TokenizerSource implementations:
The new interface CharSequenceTokenizerSource is an extension of 
de.susebox.jtopas.TokenizerSource and java.lang.CharSequence new in Java 1.4.
Two implementations are suppied: CharArraySource and StringSource. With the new
interface the expensive operation of copying characters from a String or existing
character array into the tokenizer buffer is skipped. This leads to a nice
performance boost when tokenizing String or char[] sources, depending on the 
overall amount of data that is processed.

NOTE: the two new data source classes CharArraySource and StringSource can only
be used in a Java 1.4 or higher environment!

The de.susebox.jtopas.ReaderSource has more constructors to shortcut the 
combination of constructing a java.io.Reader first and use it to construct the 
ReaderSource.

Pattern matching:
The check for non-free pattern (pattern that are bounded by whitespaces, 
separators, special sequences, comments or strings) is now performed after the 
keyword check. This could result in a different behaviour if a non-free pattern
property starts with a special sequence, string and comment start sequences or 
a separator character.

Ant build script:
The build script utilizes the macrodef-Task of Ant 1.6. Compiled classes and jar 
files are now placed under build/${ant.java.version} - e.g. build/1.4/classes 
and build/1.4/lib for javac compilers of JDK with the main version 1.4. The 1.4 
jar files can still be used in Java 1.3 VM, the 1.5 jars are for use with Java 
1.5 VM (or Java 5; You get "unsupported major.minor version" errors otherwise).

To run a junit test case, use the command: 
  ant -Dmain.class=<testcase> -Djunit.ui=<textui|awtui|swingui> run_junit

Also, compile and debug targets for the NetBeans IDE 4.0 are included (in a real
alpha state :-). 


Version 0.7
-----------
This major release contains some internal modifications, performance enhancements
and bug fixes. It is the first release where formerly deprecated classes and
methods are actually removed from the source code and the libraries.

Tokenizer cleanup:
In situations where StandardTokenizer objects are frequently created and 
discarded, they used to stay in memory since they registered with their 
StandardTokenizerProperties associate. Now, there is a close method, that should
be used like its counterparts in java.io or java.sql.

tokenizer = new StandardTokenizer(properties);
try {
  tokenizer.setSource(...);
  while (tokenizer.hasMoreToken()) {
    ...
  }
} finally {
  tokenizer.close();
}

Pattern matching:
Regular expression handling is now a lot faster than in the previous release,
mainly due to the implementation of the JDK 1.4 interface java.lang.CharSequence
as a wrapper around de.susebox.jtopas.spi.DataProvider.
Unicode-case and correct multiline handling of patterns is now available.

Special sequence matching:
Has been improved by replacing a binary search against a indexed access with the
starting letter of a sequence, comment or string.
These changes contribute to a significant performance enhancement when special
sequences, comments and strings are occuring as frequently as in programming
languages or HTML and XML.

Iterators:
While the JDK java.util.Iterator interface requires a NoSuchElementException if
next is called when there is no next element, JTopas usually returned null. This
behavior has been corrected.

ThrowableList:
The method nextException was deprecated in favour of getCause, the approbriate
JDK 1.4 method.

Performance:
Enhancements to special sequence handling and normal token handling lead to a 
performance improvement that saves around 25% and more of the time JTopas 0.6.2
needed for the same data (using a text similar to a programming language with
lots of special sequences, strings and comments and short normal token).

Version 0.6.2:
--------------
This is a maintenance release for version 0.6. See CHANGES for details.

de.susebox.jtopas.TokenizerProperties:
- Pattern handling has been improved by adding the flag F_FREE_PATTERN to
  TokenizerProperties. A pattern can now either be handled like a normal
  token that is enclosed in whitespaces, separators, comments and/or
  special sequences. Or it is handled like the latter token types that 
  can occur anywhere in the data.
- The speed of pattern matching was improved. Still, using free pattern (e.g. a
  pattern describing comments) is discouraged if there are huge amounts of data
  (large files).
- The speed of tokenizers NOT using keywords or special sequences has also been
  improved slightly.

Pattern matching:
Regular expression handling is still not perfect :-) This is mainly caused by
the absence of the JDK 1.4 interface java.lang.CharSequence in JDK 1.2 and 1.3. 
The next version of JTopas (either 0.6.3 or 0.7) will probably contain different
jar archives for the various JDK versions to solve the problem.

Version 0.6.1:
--------------
This is a maintenance release for version 0.6. There have been some bug fixes,
see CHANGES for details. The following things are important:

de.susebox.jtopas.Tokenizer:
- setReadPositionAbsolute() and setReadPositionRelative() clear the current token.
- Calls to currentToken() or currentImage() throw a TokenizerException if 
  nextToken() or nextImage() haven't been called before or if one of the 
  setReadPosition methods have been called after the last call to nextToken() /
  nextImage(). 

de.susebox.jtopas.StandardTokenizer:
- Pattern matching (regular expression) is now working.

de.susebox.jtopas.spi.PatternHandler:
- The interface was simplified. It is now equivalent to the other handler
  interfaces in the SPI package

In this version, pattern are recognized after whitespaces, separators, special
sequences, comments or strings. It is not checked, if they are followed by such
token types. Also, pattern are not recognized within "normal" token. For instance,
in "abc15def" the number pattern match 15 is not recognized. The next version
will contain means to distinguish between "separated" pattern and "free" pattern.

Version 0.6:
------------
This is a major release. The library has been largely redesigned. The former
interface de.susebox.java.util.Tokenizer was splittet into de.susebox.jtopas.Tokenizer
and de.susebox.jtopas.TokenizerProperties separating the dynamic parsing operations
from the more static token descriptions.

There are now four instead of two java archive (jar) files now (in the build/lib 
subdir): 
- jtopas.jar contains all classes
- jtopas-core.jar contains only the classes for the new tokenizer design (new jar)
- jtopas-compat.jar contains the deprecated classes of the old tokenizer design
  (new jar)
- jtopas-jt.jar contains the junit test classes

Instead of the old main class de.susebox.java.util.InputStreamTokenizer one
should use de.susebox.jtopas.StandardTokenizerProperties and 
de.susebox.jtopas.StandardTokenizer.

Along with the two interfaces Tokenizer and TokenizerProperties the classes
Token, TokenizerProperty, TokenizerException etc. were redefined in the
de.susebox.jtopas package. The old classes in de.susebox.java.util remain there,
but are deprecated.

A new, central feature of the two core interfaces Tokenizer and TokenizerProperties
is the possibility to have one TokenizerProperties instance shared between
multible Tokenizer objects. That is especially usefull in multithreaded
environments. 

The TokenizerProperties interface has now an event listener scheme. 
TokenizerEventListener implementations can register and deregister themselves
to receive notification about properties added, removed or changed in the
TokenizerProperties.

The communication between a Tokenizer and its backing TokenizerProperties can
be realized through the interfaces defined in the de.susebox.jtopas.spi package.
These interfaces replace functionality of the old PluginTokenizer that is
now deprecated along with its supporting classes and interfaces.

The new token type PATTERN is now available. A pattern is a regular expression
used to scan the input. Currently, the backing software is the java.util.regex
package of JDK 1.4. For compatibility with JDK 1.3, the pattern handling is
realized by dynamic invocation of the java.util.regex classes, therefore not very
fast. Also, there is no way to detect an incomplete pattern, practically reducing
the pattern matching ability of the StandardTokenizerProperties to "short" pattern.
