Coverage-based Testing of C Objects

After all spec-based tests have been created and encoded in STImpL specifications, and all script details have been worked out, it is time to proceed with code-based testing.

To do this, begin with your base version. Create a branch trace instrumented version of it, create a trace script for your STImpL test suite(s) for it, and obtain the traces for all tests in the STImpL test suite(s). Use the th_builder to construct a test history for those traces.

Now, run the Aristotle tool coverage to see what has been covered or remains to be covered in your program. To run this you type:

coverage <test-history-file> <prog-name> [ALL|UNCOV]

where the first argument names your test history file, and the second names your program with the .c suffix. The third argument is optional, if present and ALL the tool reports coverage data on all branches, if absent or UNCOV the tool reports on just branches not yet covered.

When you type:

coverage tracedir/th calc.c ALL

here is a portion of what you see:

__________________________________________________

Function: pop

 edgeid  coverage status   sourceline,edgelabel
---------------------------------------------------
     1        HIT                 107,
     4        HIT                 110,T
     5        NOT HIT             110,F


Function: push

 edgeid  coverage status   sourceline,edgelabel
---------------------------------------------------
     1        HIT                  94,
     3        HIT                  95,T
     4        NOT HIT              95,F
__________________________________________________

If you had left out the ALL or typed UNCOV. the lines labeled HIT would be absent from this.

The information lists edge ids, the coverage status of the edge, and the sourceline,edgelabel corresponding to that edge.

Using the src_printer tool, typing src_printer calc.c, you can get a print out of the calc program listing line numbers, which you can cross-reference with the above information. Here's a portion of it:

92     void push(f)
93     double f;
94     {
95          if (sp < MAXVAL)
96          {
97             val[sp++] = f;
98          }
99          else
100         {
101             printf("error: stack full\n");
102             clear();
103         }
104    }
105

You can see that the coverage information tells you where the branches are relative to this source listing, with labels indicating which direction of flow from the branch is involved. Edge labels out of cases are listed too.

Using this tool you should be able to accomplish the next task. That task is to find test cases to cover all executable branches, with some exceptions. As you find test cases, start creating them, make a new STImpL file to contain them (universe.cov) so we know which tests are which. You'll need to reexecute your tests occasionally to see whether a test case covers more than just the branch you were aiming at.

In addition to new test cases, this process will also yield a list of branches that are not and will not be covered by us. Keep a listing of these, with information on why they won't be covered. Save this information in a file called "Uncovered_Branches.info" and save this file in the "info" directory of the object directory structure.

Reasons to not cover a branch:

  1. If a branch cannot be executed by any input to the program, then it is nonexecutable. Reason = "nonexecutable".
  2. If a branch can be executed only with great difficulty by making some error happen (e.g. making a malloc fail) we'll choose to leave it unexecuted. Reason = "requires failure of xxx, decided not to cover."
  3. If we have made some decision about your program of the form "we will ignore that functionality in our testing", and the branch can only be executed if we do not ignore that functionality. Reason = "left uncovered due to decision not to address this portion of the program's functionality."

Finding inputs to execute branches is difficult. Determining non-executability is difficult. In general doing this will require you to become more familiar with the program and its code. Suggestion: start with code near the "top" and "beginning" of execution, where conditions for execution are easier to determine. Executing some of this code will also cover code later down.

Note that the TSL tests for our medium-size C objects are functional tests. and these objects do not contain enormous numbers of coverage tests. In this sense the objects are not un-realistic since in practice full coverage is often not achieved. But our process faced the same cost-benefits tradeoffs faced in practice: tester time versus test effectiveness. Our object creators worked on coverage testing for a time limited by our budget.