Creating a Java Fault Matrix

A fault matrix file contains information about the faults exposed by the test cases in a STImpL file for a particular version of an object. It gives mappings between faults and the test cases that expose them. The steps to generate a fault matrix file are given below.

Steps to generate a fault matrix file:

  1. Make sure that the javamts and gen_fault_matrix tools are present in the current path. To download these tools see: tools
  2. Set enviroment variable experiment_root to point to the parent directory of the object's directory by typing setenv experiment_root <path> For example, suppose your object directory, say siena, is located at "/cs_machine/exp_group/experiments", then the experiment_root is set by typing setenv experiment_root /cs_machine/exp_group/experiments.
  3. Run the tool to create the fault matrix file by typing: 
    make_fault_matrix_with_names.sh <program object> 
    <installation script> <version start> <number of versions> 
    <STImpL path> <test STImpL file> <executable file>
    <fault matrix file> [<line_adjust>]

    <program object> : name of program object directory.
    <installation script> : name of object installation script, which resides in "$experiment_root/object/scripts".
    <version start> : the version starting from which we want to create a fault matrix file.
    <versions> : the total number of versions for which we want to create fault matrix files.
    <STImpL path> : path name which contains STImpL files (normally, "testplans.alt", see the README in the Java Tools --> package for an exception).
    <test STImpL file> : name of the test STImpL file (either .tsl.universe or .cov.universe file)
    <executable file> : the main program to be run and input for javamts. If you run testdrivers instead of the main program, then you should give NULL for this parameter.
    <fault matrix file> : name of the fault matrix file to create.
    [<line_adjust>] : optional argument (no value or "1" is given). See NOTE at the end of this document for more details.

We will look at the siena object as an example.

In siena, faults are seeded from version v1 onwards and the STImpL file v0.tsl.universe is inherited accross all the versions, i.e. from v1 to v7.

The name of the fault matrix file to create is fault-matrix.v0.tsl.universe (described in the Java object directory specification). To run the tool we type:

make_fault_matrix_with_names.sh siena install_seeded.sh 1 7 
    testplans.alt v0.tsl.universe NULL fault-matrix.v0.tsl.universe 1

This command creates the fault matrix files for the test STImpL file v0.tsl.universe on versions v1 to v7 and stores them in the object's info directory, under subdirectories v1 to v7.

Note that to create a fault matrix, this tool executes each version on each test in the STImpL file and this may take some time!

NOTE: When and why you need the <line_adjust> option

When you seed faults in a Java source file and preprocess it (cpp) in order to run that faulty version, it may cause differences in the total number of lines or shift lines between the original and faulty versions. Normally this isn't an issue, but test cases that print exception stack traces may report false-positive output differences when comparing outputs between the original and faulty versions. This occurs when a fault does not affect the output of a test case semantically, but causes a change in the line numbers printed in a stack trace.

To prevent this problem, the line_adjust options applies a utility to equalize line numbers between the original and faulty versions. To use this option, your installation script must be modified to handle this equalization of line numbers. That is, the initial .cpp files containing all of the fault-ids (where no faults are turned on yet) also need to be line adjusted (this is necessary to handle the case where an injected fault inserts new code into the object).

An example of the use of the line adjustment option can be found in the Java Tools package.