Java Object Preparer's Handbook

Table of Contents

  1. Overview
  2. Object Selection
  3. Object Organization
  4. Test Creation
    1. TSL-based Testing
      1. TSL Specification
      2. TSL Test Creation
    2. Coverage-Based Testing
    3. JUnit-Based Testing
    4. Test & Script Validation
  5. Gathering final test histories and test timings
  6. Fault Localization
  7. Generating Fault-Matrices

1. Overview

This document describes the steps we follow to set up a Java program as an experiment object. It is written primarily as a set of instructions to persons who set up these objects, and who continue to set up others, but it is also valuable as an aid to understanding the object setup, and choices made in that setup. Also be aware that some aspects of the preparation process vary depending on the types of analyses and experiments for which the object is expected to be of primary interest.

2. Object Selection

We seek Java objects with qualifications as described in:

object-requirements

Initially, we want objects that definitely meet our 1st-tier required-qualities, and that appear to meet (or can't be determined not to meet) our 2nd-tier required-qualities.

After we have a collection of these, we determine which of these are worth considering further to see if they do in fact meet the 2nd-tier requirements.

Completing this task involves downloading objects that look like they have potential, and saving any objects that look like candidates in well-named directories. Completing this task requires downloading the objects.

Also part of this task is ensuring that you can build and run all versions of the program that we are considering using, and seeing what sort of output is involved. We need outputs that can be easily automatically checked by comparing with saved outputs. So, programs that manipulate the screen or produce graphical output where we'd have to save screen-dumps are not good candidates. Programs that produce text output to the screen and/or output to files are fine.

Concurrency

Programs that make use of concurrency (threading) may be of interest for experimentation involving analysis techniques related to validation and verification of such programs. Such programs were generally not suitable for many of our program analysis techniques (for single threaded programs), principally because those techniques depend on deterministic outputs. Determinism of output remains a primary concern for any testing object and sources of non−determinism should still be eliminated as much as possible (see item 5.c below.) However, recent efforts in concurrency analysis frameworks like JavaPathFinder have allowed inclusion of concurrency testing artifacts. As the art of concurrent testing matures we expect more concurrency test subjects to be made available.

3. Object Organization

After selecting objects we organize them into a format that supports later experimentation.

We have an agreed-on format described in:

java object directory specification

Read this format file and look at existing objects for examples. You'll be starting by working in the "versions" area, then later in "inputs" and "testplans" and "scripts", as we get to later tasks.

Note: It is important that you set the environment variable "experiment_root" before working with any of the SIR objects. This variable should provide the absolute path to the directory into which you have placed the SIR object directories. This enables scripts and tools to resolve needed paths without hardcoding system specific paths.

4. Test Creation

Various testing mechanisms could be supported. We intend to begin with TSL-based tests, code coverage-based tests, and JUnit tests.

  1. TSL-Based Testing
    1. TSL Specification Creation

      TSL is a "Test Specification Language" used to write spec-based tests for a product. When you write a test spec in this language, we have a tool that translates your spec into "test frames", from which you then generate tests.

      To accomplish this task you should first read a paper on TSL: "The Category-Partition Method for Specifying and Generating Functional Tests", Thomas Ostrand and Marc Balcer, Communications of the ACM, Volume 31, Number 6, June 1988, pp 676-686. (The paper is available in the ACM Digital Library online).

      Next, read the documentation on our TSL tool and TSL format (which is almost the same as the format described in the paper). This is contained in the file:

      tsl-specs

      What you need to do is write a TSL spec for your base version, based on its documentation. (Or perhaps this is multiple TSL specs, since each spec corresponds to some portion of the functionality of the system.)

      You need to ensure that this spec results in a sufficient but not excessive number of test frames. (For our 5 - 15K line programs, we decided that 200-500 frames was the target range.) Add constraints if necessary to reduce the number of frames.

      Subsequent versions should be able to "inherit" this specification, or most of it, and may need additional tests based on new functionality, which can be encoded in an additional specification added to that version, or in a further refined TSL spec.

    2. TSL Test Creation

      Creating test cases involves determining what inputs and input files are required, setting up those input files, and encoding the tests in a "STImpL File". A program called "Java-make-test-script" (javamts) transforms STImpL files into various types of scripts that can be used to automatically execute the tests or gather traces for the tests.

      The format of the STImpL file, the types of scripts created by running javamts, and the usage of javamts, are described in file:

      java-stimpl-file-spec

  2. Coverage-Based Testing

    Σοφία (Sofya) is a publicly available system for analysis of Java that can be used to determine basic block coverage and create coverage adequate test suites. Please refer to the Σοφία coverage tool documentation for additional information.

  3. JUnit-Based Testing

    In addition to traditional types of test suites, for some Java objects (ant, xml-security, jmeter, and jtopas), we provide JUnit test suites. JUnit is is a new Java testing framework, that allows automation of tests for classes, and that is increasingly being used with Java systems. JUnit test suites have been supplied with each Java program from its open source software host.

    Test cases are encoded in a STImpL file whose format is same as TSL STImpL file format.

  4. Test and Script Validation

    After you have created a set of TSL tests or the coverage based tests, you should perform several validation tasks, that will help us determine if there are problems with the objects or tools. Details on the tasks are given in file:

    test-and-script-validation

    Note that, as detailed in step 2 of these instructions, this may reveal sources of nondeterminism. These need to be made deterministic. For details see file:

    nondet-stderr

5. Gathering final test histories and test timings

After performing all validation tasks for your tests, test histories and timings can be gathered. We'll be creating tools for this and will make them available and document their use here.

6. Fault Localization

Next, we need to find or create faults. There are two approaches for doing this in the objects:

  1. Find Existing faults, as described in:

    existing-faults

  2. Seed faults, as described in:

    fault-seeding

7. Generating Fault-Matrices

After seeding faults in the versions, generate fault-matrices summarizing the relationship of test cases to faults, as described in:

fault-matrices.