Java Object Handbook
Table of Contents
1. Overview
This document describes the organization of a Java program as an experiment object. Be aware that some aspects of the preparation process vary depending on the types of analyses and experiments for which the object is expected to be of primary interest.
The objects in the repository have been designed with support of a certain experimentation workflow in mind. However, we have attempted to create a design that is generic enough to allow customization to other workflows as appropriate. One approach to conducting an experiment with the infrastructure is the following:
- Download or create a new copy of one or more objects in a local space for the experiment.
- Set the environment variable "
experiment_root" to reflect the location of the experiment object. This variable should provide the absolute path to the directory into which you have placed the SIR object directory. This enables scripts and tools provided by the infrastructure to resolve needed paths without hardcoding system specific paths. - Determine the artifacts that you want to utilize, such as specifications, versions, seeded faults, and inputs.
- Create scripts that can leverage the common organizational structure of the objects to perform the desired experimentation tasks on a version of the object, and that can further leverage the infrastructure by relying on common locations for outputs.
- Finally, create a top-level script that will perform the experimentation tasks across all versions of an object.
In this way, the object infrastructure facilitates a high degree of automation, enabling experiments that can more rapidly gather much larger amounts of reliable data, in turn leading to results with stronger claims of significance and fewer threats to validity. In some experiments, one can even consider a master script that automatically executes the experiment tasks across multiple objects.
SIR provides a number of tools to help you conduct experiments on the object programs. The above is only one example of a process for using the infrastructure; many others are possible. However, regardless of the process you choose, you should always perform step 2 above if you intend to use the tools provided by the infrastructure.
2. Object Selection
To populate the repository, we sought Java objects that satisfied various guidelines.
They needed to be "real" programs, in the sense that they are non-trivial
programs created by experienced software programmers to implement solutions to actual
problems. In other words, we did not consider programs created specifically
for research or experimentation, or programs created by students. We required
the objects to have a "sufficient" number of classes (historically
at least 10), with free and distributable source code. There needed to be at
least 5 versions available, which could include versions identified by source
control tags where available. Finally, we preferred programs that could be built
from their source code and run on Solaris. Where possible, we tried to find and
include other associated artifacts, such as existing specifications, test suites,
and fault reports. We have also placed a priority on obtaining objects that
provide existing test suites, especially JUnit test suites.
In some cases we also sought objects that can be separated into
distinct component and application modules. These objects facilitate
investigation of component-based techniques. In such cases, we required
that at least three seperate interactions could be found between the application
and components. Examples of this type of object are nanoxml and
siena.
We limited our selection of programs to those that generate outputs that can easily be automatically checked by comparison with saved outputs. Programs that manipulate the screen or produce graphical output such that it is necessary to save screen-dumps were not considered suitable and are not found in the repository. Programs that produce text output to the screen and/or output to files were accepted.
Concurrency
Programs that make use of concurrency (threading) are of interest for experimentation involving analysis techniques related to validation and verification of such programs. However, such programs are generally not suitable for many of our present program analysis techniques, which are designed for single threaded programs. Particularly affected are techniques that depend on the ability to compare deterministic outputs. Note that historically we have preferred deterministic programs (or those that can be made so relatively easily), however, we expect to begin including concurrent objects as the repository evolves and expands.
3. Object Organization
Objects in the repository are organized into a format that supports experimentation activities, which is described here:
Java object directory specification
4. Test Suites
Various testing mechanisms are supported to varying degrees by the objects in the repository. The principle testing techniques presently available are test specification language (TSL) based testing (a functional test specification technique), and JUnit testing.
- TSL-Based Testing
- TSL Specification
TSL is a test specification language used to write specification-based tests for a product. The repository provides a tool that that translates a TSL specification into test frames, which can then be used to generate actual tests.
Existing TSL specifications are provided with a number of the objects in the repository. These specifications were written to produce a sufficient but not excessive number of frames, defined at the time as 200 to 500 frames. If you simply want to generate the test frames from these existing specifications, you should refer to the instructions for using the tsl tool provided by the repository:
Should you wish to modify these specifications or create new specifications for objects in the repository, you will first want to familiarize yourself with how TSL specifications are constructed. The process is described in the following paper: "The Category-Partition Method for Specifying and Generating Functional Tests", Thomas Ostrand and Marc Balcer, Communications of the ACM, Volume 31, Number 6, June 1988, pp 676-686. (The paper is available in the ACM Digital Library online). You should then read the documentation describing the specifics of the TSL format that is processed by the tsl tool available from the repository (which is almost the same as the format described in the paper):
- Test Case Creation
Repository objects that provide TSL specifications also provide STImpL files that encode descriptions of how test frames are to be realized as actual executable tests, and sets of inputs required to support those tests. A program called
javamts(Java make-test-script) is provided to translate these STImpL files into various types of scripts that can be used to automatically execute the tests and gather test traces. If you simply want to generate test scripts from these existing STImpL files, you should refer to the instructions for usingjavamts:Should you wish to modify or create new STImpL files, the STImpL file format is described here:
- TSL Specification
- Coverage-Based Testing
Σοφία (Sofya) is a publicly available system for analysis of Java that can be used to determine basic block coverage and create coverage adequate test suites. Please refer to the Σοφία coverage tool documentation for additional information.
- JUnit-Based Testing
Some Java objects in the repository (currently ant, xml-security, jmeter, and jtopas), come with JUnit tests instead of the traditional functional TSL tests. JUnit is a new Java unit testing framework that allows automation of tests for classes, and that is increasingly being used for testing Java software. The JUnit test suites that are currently available were obtained with each Java program from its open source software host.
For these objects, test cases have been automatically extracted from the JUnit test suites and encoded in a STImpL file in the same way as the TSL tests. These STImpL files are not necessary to run the provided JUnit tests (and indeed yield poorer test suite execution times), but may be of use to certain analyses.
5. Fault Seeding
Seeded Faults
As indicated previously in the Java object directory specification, all of the Java objects currently available from the repository are provided with seeded faults. Each version of an object found in the "seeded" directory in versions.alt contains one or more ".java.fault" or ".cpp" files that are equivalent to the matching ".java" files, but with seeded faults inserted and guarded by preprocessor conditionals. (We keep both clean and fault seeded versions of modified files). Additionally, there may be .h files that contain the preprocessor directives needed to activate specific faults in packages. There will always be a FaultSeeds.h file that contains all of the "#define" declarations that can be used to activate individual faults in the object. Faults can then be activated using either the EqualizeLineNumbers Java class which is provided by the java_tools package, or by using the C preprocessor, cpp. Files with faults inserted typically are named with a .cpp filename extension which replaces the .java extension of the faulted subject Java source file. Because EqualizeLineNumbers will generate source (Java) code that have the same code on the same line numbers for unfaulted as well as faulted versions, we highly recommend using EqualizeLineNumbers instead of the cpp approach described below. We describe both fault insertion methods here for completeness.
Example: to enable the first fault within the The Sample.cpp file, which contains the faults embedded in cpp directive statements, the syntax for using the EqualizeLineNumbers class to activate the first fault within the file (which is how EqualizeLineNumbers refers to faults by the order of their appearance in the source file) the command would be:
java EqualizeLineNumbers Sample.cpp 1 Sample.java
When using EqualizeLineNumbers to activate faults, you should install the fault seeded version and activate fault 0 (zero) when you want to run/test an unfaulted version of the java source. Doing this will ensure line numbers agree between faulted and unfaulted versions and eliminates one source (line number differences) of non-determinism in the test outputs.
If you want to activate the fault with fault-id F_AG_1 in the Sample.java.fault file using the cpp preprocessor, you would type:
cpp -D F_AG_1 -P -C Sample.java.fault Sample.java
Now Sample.java can be compiled along with the other unaltered files.
Using the cpp method can result in non-deterministic output due to line numbers reported on the output, e.g. within Java Exception error output.
To facilitate disclosure of experimental processes and support claims of validity, the process used to inject these seeded faults is described in the following document:
Fault Matrices
Objects are also provided with fault matrices that summarize the relationship of test cases to faults (e.g. they correlate faults to the test cases that reveal those faults). We do not guarantee these fault matrices as authoritative for any platform or environment other than the one under which they were generated. For this reason, we generally encourage users to generate their own fault matrices, a process that is described in the following document:
Notes on interpreting the fault matrices can be found in the fault matrix interpretation document.