Fault Matrix Interpretation
The fault matrices delivered with SIR objects generally describe which test finds which fault. A typical fault matrix file contains three sections of interest, these are an overall report section, a test input/test frames section, and the fault matrix section.
It should be noted that not all SIR subjects provide a fault matrix. Those that do not either were never originally delivered with one or a fault matrix is not applicable because the object is used for a different type of testing experiment. Fault matrices are only applicable to SIR subjects that attempt to do fault insertion or express mutation faults for testing analysis.
Overall Fault Matrix Report Section
The first section of a fault matrix file contains a pair of lines that tell
how many (faulted) versions are reported on in the fault matrix as well as the
total number of tests that were executed on each. An example of this section
(taken from the tcas subject fault-matrix):
41 listversions
1608 listtests
This indicates that 41 faulted versions are reported on in this fault matrix
and 1608 tests were run on each faulted version.
Test Inputs/Test Frames Section
Following the report section lines is the test inputs/test frames block.
The contents of this block are simply the contents of the universe
file that was used to create the test script (if the mts or
javamts tool was used) or the contents of the shell script
that performs the tests on the subject. For the tcas subject
the first few lines of this section looks like this:
958 1 1 2597 574 4253 0 399 400 0 0 1 627 0 0 621 216 382 1 400 641 1 1 0 549 1 1 4398 133 1445 1 641 639 0 0 1 576 0 1 3469 183 381 2 641 501 1 0 1 992 1 0 3342 23 4657 1 640 741 0 0 0 548 0 1 34 542 3514 2 499 401 1 1 1 ...Since the
tcas subject uses the contents of the universe file
as inputs, each line represents a test case (frame).
For other SIR subjects, this section may appear differently. The mts
and javamts tools are used to compose test cases from universe
files. When this is the case, this section will have the appearance:
-P [--quiet /home/ugrad/kmurphy/frost/worksandbox/sed/inputs/default.in] -C [t1: "quiet" output] -P [--silent /home/ugrad/kmurphy/frost/worksandbox/sed/inputs/default.in] -C [t2: "silent" output] -P [-V /home/ugrad/kmurphy/frost/worksandbox/sed/inputs/default.in] -C [t3: test version (-V) output] -P [--version /home/ugrad/kmurphy/frost/worksandbox/sed/inputs/default.in] -C [t4: test version (--version) output] -P [/home/ugrad/kmurphy/frost/worksandbox/sed/inputs/default.in] -C [t5: no command source given] -P [-e 's/dog/cat/' -e '/cat$/d' /home/ugrad/kmurphy/frost/worksandbox/sed/inputs/default.in] -C [t6: multiple -e commands] ...as in this example from the universe file of the
sed SIR subject. Each
line in this sequence again represents a test case. See the mts tool
for more information on this format and its interpretation.
Fault Matrix Section
Following the test cases section is the actual fault matrix data. The formatting
of this section is straightforward but may not be obvious to the first time
user. Paramount to understanding the matrix is the concept of the "test" specific
view. The test specific view expresses the results of a test across all versions
that test was performed upon. If we examine the first part of this section in the
fault matrix for the sed subject we see this:
unitest0:
v1:
0
v2:
0
v3:
0
unitest1:
v1:
0
v2:
0
v3:
0
unitest2:
...
unitest5:
v1:
0
v2:
0
v3:
1
...
The line containing the unitest0: string indicates the beginning
of the report for test number 1 (since these are 0 significant representations).
The next line, v1: indicates that what follows is the results
of a comparison between the normal (unfaulted) outputs of test 1 with output
of the faulted version (variant) 1. When a difference is detected between the
outputs (as reported by the cmp command) the line following this
v1: line will contain a 1 as shown above for unitest5 v3
(faulted variant 3.)
If no output change was detected
between the original and faulted versions this next line contains a 0 as
seen for unitest0 v1: above.
Similarly, the v2: line introduces the results of an output comparison
between the original test output and the output of the test when it was run
on version (variant) 2. Since sequential version 1 of the sed SIR subject
only contains 3 faults, there are only 3 of these fault-version and result line
pairs for each test (unitest).
Since each SIR subject typically has several sequential versions that have faults applied to each, the info directory is arranged with subdirectories for each of these versions and within that directory a fault-matrix file corresponding to the universe file will be found.