Software-artifact Infrastructure Repository

• Home
• Download Objects
• Download Tools
• C Object Handbook
• Java Object Handbook
• Java Invariant Checking
  Examples Handbook
• SPL Object Handbook
• Concise Object Handbook
• C Object Preparer's Handbook
• Java Object Preparer's Handbook
• SPL Object Preparer's Handbook
• Concise Obj. Preparer's Handbook
• Reporting Experimental Results
• Citing SIR
• SIR Users and Publications
• Frequently Asked Questions
• Acknowledgments

Seeding Faults

The difficulty of locating existing faults has led us to seed faults. Another motivation for doing this is that most of our work involves regression testing --- looking for errors caused by code changes. For our purposes we need to seed regression faults. The following process could, however, be adapted to seed general classes of faults.

Process

1. Placing faults in the code

Fault Seeders are programmers with at least two years of programming experience in C/C++. At least two faults seeders should work on each object.

1.1. Finding the changes between versions

With the assistance of a diff tool, determine where the changes occurred. We recommend using diff -u (unified diff) in order to get more accurate differences. Be aware that if the changes between versions were large, the diff tool may carry differences forward, even when the code is identical. You should verify the differences independently and use diff only as a guide.

1.2. Introducing the fault

Each programmer must do this independently to make the process as objective/credible as possible.

1.2.1. Type of fault

In your life as a programmer, you have inserted and (hopefully) removed many type of faults. This fault seeding process needs to mimic what you have found during your experience in terms of type and number of faults. Here is a simple fault classification scheme that can be helpful for thinking about different types of faults:

1. Fault associated with variables: definition of variable, redefinition of variable, deletion of variable, change value of variable in existing assign statement.
2. Fault associated with control flow: addition of new block of code, deletion of path, redefinition of execution condition, removal of block, change order of execution, new call to external function, removal of call to external function, adding function, removing function.
3. Fault associated with memory allocation: allocated memory not freed, not initialized, erroneous pointer.

Although faults are often more complicated than the ones just mentioned, complex faults can often be decomposed into these simpler types of faults. For the integrity of experiments, it is very important that you try to include different types of faults to simulate a real fault insertion process.

1.2.2. Fault location

A regression fault can be located only within the changes between versions (limited by the differences found in 1.1). We will be assuming then, that the programmer that made the modification inserted a fault (although we will be modifying the modified code to simulate that). There is no restriction on the size or type of fault at this stage. The only requirement is that the changed program can still be compiled and executed.

1.2.3. Code change

Each target section of code (line, block, etc) where a fault is meant to be seeded needs to be duplicated (copy and pasted). One copy will be the original, the second copy is the one with the embedded fault. The copies need to be enclosed in preprocessor directives as follows:

#ifndef FAULTY-fault-id

    // original code

#else

    // code with the embedded fault

#endif

The fault-id format is defined in the next section.

In addition, the following line should be included at the top of the file:

#include FaultSeeds.h

Make sure the code compiles after you have made your changes. First, create an empty FaultSeeds.h file, compile and link. The program should behave just as the original program did. Then, add the following line to the header file: #define FAULTY-fault-id, compile and link to test one of your modifications. Change the fault-id in the define line to test each of your modifications individually.

1.2.4. Format of fault-id

For each fault, you need to keep a record that includes:
Fault ID, baseline version, the ID # of the associated difference, a description of the introduced fault (including a link to the classification). Each fault ID must follow the following format: F_FaultSeederInitials_Number.

Examples:

F_GR_1 (first fault found by G.Rothermel), 
F_SE_5 (fifth fault found by S. Elbaum)

1.2.5. Iterate

We'd like to begin with approximately 20 potential faults per version, but this requires adjustment for scantily modified versions. Thus, a total of Z faults (Z = MIN (number of diffs in version / 2, 10)) should be generated per version. In other words, you will seed 10 faults per version or less if the number of differences is smaller than 20. Repeat the process in 1.2 until Z faults are found.

2. Selecting faults to keep for experiments

2.1. Common fault filtering

Since two programmers do the seeding process independently, some faults may be repeated. Repeated faults will be removed after all Z faults have been inserted. Although the programmers could work together to avoid overlapping, this would hurt the validity and credibility of the process so we decided against it.

The two programmers can perform this process by looking at the fault record, referring to the code when necessary to compare seeded faults that seem to be similar. If two seeded faults are identical, just add DUP to the fault record of one of the faults.

2.2. Exposure filtering

The modified code from both programmers needs to be joined. Note that only one original section is necessary per program. This should be a simple and short (maximum 10 modifications) cut/paste process but needs to be done carefully. Make sure you compile, link and test your program like you did in 1.2.4.

Then, run the test suite on each version to proceed with the filtering. We will filter out two types of faults:

2.2.1. Faults that are exposed by more than X% of the tests.

These faults are removed because they are considered too easy to find (are unlikely to be introduced by an experienced programmer, and if they are introduced, are likely to be detected during unit testing). If a seeded fault is exposed by more than X percent of the tests, then add easy to the fault record of the faults.

2.2.2. Faults that are not detected by any tests.

There are two alternatives depending on time availability. If developing a test to expose the fault would take less than Y hours, then write the new test and verify that it works. If developing a test takes more than Y hours, the write an N-EXP next to the fault description.

Ultimately after filtering we want 3-10 faults per version. Note that we can include only one fault per line changed.

2.3. Final Setup

Keep in the header file only the fault-ids that were not filtered. Test the program and then, move on to the next version!!!

3. Additional Clarifications

It is our intent that seeded faults be unique not only within, but also across, object versions, and fault seeders should verify that this is intent is satisfied. Fault names, however, are unique within versions only, not between them.

Because we are interested in regression faults, our fault seeding focuses on the versions after, but including, v0. These changed versions can contain regression faults. As noted earlier, additional, non-regression faults could be seeded in all versions. (For example, the Siemens programs contain "general" faults, not regression faults.)

On some object versions it may be the case that no regression faults were successfully seeded; typically this occurs in cases where there are few changes made to the code for versions. But these versions have been retained, since they may be useful for other purposes.