Software-artifact Infrastructure Repository

• Home
• Download Objects
• Download Tools
• C Object Handbook
• Java Object Handbook
• Java Invariant Checking
  Examples Handbook
• SPL Object Handbook
• Concise Object Handbook
• C Object Preparer's Handbook
• Java Object Preparer's Handbook
• SPL Object Preparer's Handbook
• Concise Obj. Preparer's Handbook
• Reporting Experimental Results
• Citing SIR
• SIR Users and Publications
• Frequently Asked Questions
• Acknowledgments

Seeding Java Faults

1. Placing regression faults in the code

Regression faults are faults in changed code, and these are the faults in which we are interested and describe here. (Other types of faults could be inserted following a similar process.)

Fault Seeders are programmers with at least two years of programming experience in Java. Ideally, at least two faults seeders work on each object.

1.1. Finding the changes between versions

With the assistance of a diff tool, determine where the changes occurred. We recommend using diff -u (unified diff) in order to get more accurate differences. Be aware that if the changes between versions were large, the diff tool may carry differences forward, even when the code is identical. You should verify the differences independently and use diff only as a guide. Also, if available, you can use adiff (this can be downloaded from href="https://sir-public.github.io/SIR/tools.html">tools) which produces a list of changed functions.

1.2. Introducing the fault

Each programmer must do this independently to make the process as objective/credible as possible.

1.2.1. Type of fault

In your life as a programmer, you have inserted and (hopefully) removed many type of faults. This fault seeding process needs to mimic what you have found during your experience in terms of type and number of faults. Here is a simple fault classification scheme that can be helpful for thinking about different types of faults:

1. Faults associated with variables: definition of variable, redefinition of variable, deletion of variable, change value of variable in existing assign statement.
2. Faults associated with control flow: addition of new block of code, deletion of path, redefinition of execution condition, removal of block, change order of execution, new call to external function, removal of call to external function, adding function, removing function.
3. Faults associated with specific Java language constructs or facilities (such as constructors or inheritance).

Although faults are sometimes more complicated than the ones just mentioned, complex faults can be decomposed into these simpler types of faults. For the integrity of the experiment, it is very important that you try to include different types of faults to simulate a real fault insertion process.

1.2.2. Fault location

A regression fault can be located only within the changes between versions (limited by the differences found in 1.1). We will be assuming then, that the programmer that made the modification inserted a fault (although we will be modifying the modified code to simulate that). There is no restriction on the size or type of fault at this stage. The only requirement is that the changed program can still be compiled and executed.

1.2.3. Code change

Each target section of code (line, block, etc) where a fault is meant to be seeded needs to be duplicated (copy and pasted). One copy will be the original, the second copy is the one with the embedded fault. The copies need to be enclosed in preprocessor directives as follows:

#include FaultSeeds.h
	:
#ifdef fault-id

	// code with the embedded fault

#else

	// original code

#endif

(Yes, these are C directives, but we'll be using the C preprocessor on fault seeded files, as described later.)

Also, the fault-id format is defined in the next section.

In addition, the following file should be created to keep fault-id information:

     FaultSeeds.h

This file will be used by the fault-matrices generator, to map faults to the test cases that reveal them.

To test a faulty version, you need to turn one of the faults on before compiling faulty version. The c preprocessor (cpp) performs this process.

Example:

Assume you have two java files to be fault seeded: A.java and B.java. Copy A.java (B.java) into A.cpp (B.java) and keep A.java (B.java) unmodified.

After you seeded faults, A.cpp has fault-id, F_A_HD_1 and F_A_AK_1, in order, and B.cpp has F_B_HD_1, F_B_HD_2, and F_B_AK_1 in oder. That is, F_A_HD_1 appears earlier than F_A_AK_1 in A.cpp file.

FaultSeeds.h contains all faults that reside in an object version in the following format:

[fault_id] [fault_order_in_file] [fault_file_name]
[fault_id] [fault_order_in_file] [fault_file_name]
[fault_id] [fault_order_in_file] [fault_file_name]
    :

Using the example, FaultSeeds.h should be:

F_A_HD_1 1 A.cpp
F_A_AK_1 2 A.cpp
F_B_HD_1 1 B.cpp
F_B_HD_2 2 B.cpp
F_B_AK_1 3 B.cpp

There are NO empty lines, each line corresponds to exactly one fault.

1.2.4. Format of fault-id

For each fault, you need to keep a record that includes: Fault ID, baseline version, the ID# of the associated difference, a description of the introduced fault (including a link to the classification). Each fault ID must follow the format: F_ClassFileInitials_FaultSeederInitials_Number.

Examples:

F_NV_GR_1 (first fault in Nonvalidator.java found by G.Rothermel),
F_SP_SE_5 (fifth fault in StdXMLParser.java found by S. Elbaum)

1.2.5. Iterate

We typically aim to begin with approximately 20 faults per version.

A total of Z faults (Z = MIN (number of diffs in version / 2, 10)) should be generated per version. Thus you will seed 10 faults per version or less if the number of differences is smaller than 20. Repeat the process in 1.2 until Z faults are found.

2. Selecting faults to keep for experiments

2.1. Common fault filtering

Since two programmers do the seeding process independently, some faults may be repeated. Repeated faults will be removed after all Z faults have been inserted. Although the programmers could work together to avoid overlapping, this would hurt the validity and credibility of the process so we avoid this.

The two programmers can perform this process by looking at the fault record, referring to the code when necessary to compare seeded faults that seem to be similar. If two seeded faults are identical, just add DUP to the fault record of one of the faults.

2.2. Exposure filtering

The modified code from both programmers needs to be joined. Note that only one original section is necessary per program. This should be a simple and short (maximum 10 modifications) cut/paste process but needs to be done carefully. Make sure you compile, link and test your program as you did in 1.2.4. Then, run the test suite on each version to proceed with the filtering. We will filter out two types of faults:

2.2.1. Faults that are exposed by more than X% of the tests.

These faults are removed because they are considered too easy to find (are unlikely to be introduced by an experienced programmer, and if they are introduced, are likely to be detected during unit testing). If a seeded fault is exposed by more than X percent of the tests, then add easy to the fault record of the faults.

2.2.2. Faults that are not detected by any tests.

There are two alternatives depending on time availability. If developing a test to expose the fault would take less than Y hours, then write the new test and verify that it works. If developing a test takes more than Y hours, the write an N-EXP next to the fault description.

Ultimately after filtering we want 3-8 faults per version. Note that we can include only one fault per line changed.

2.3. Final Setup

Keep in the header file only the fault-ids that were not filtered. Test the program and then, move on to the next version!