Test and Script Validation

Either after an object is done, or as you complete the work on each version, complete the following test and script validation tasks on the universe files present in the version(s).

  1. To make sure we can automatically run all tests and save their outputs, for use in automated testing and output validation, on each version:
    • run MTS creating a runall script,
    • run the script and verify that it executes all your tests, saving outputs as it should
      • initially try "runall" with output going to your screen, and see if anything other than ">>>executing test k" comes out.
      • next try "runall" redirecting output to a "runall.log" file and see if anything gets output to the screen. It shouldn't. Check the log after and see if anything seems odd.
      • after running the version, save the outputs directory someplace, let's say in an "outputs" subdirectory you'll create in the directory for that version.
  2. To further test our ability to automate test execution with output differencing:
    • run mts creating a runall-and-diff script, (ask it to do "differences" against the output directory you saved previously for that version)
    • run the script created:
      • verify that it executes all your tests, saving outputs as it should
      • verify that it reports no differences
      • try this with the output-to-screen and output-redirected approach

    (The following steps assume the use of Aristotle to measure coverage, similar steps could be taken if other coverage tools are used.)

  3. To test whether instrumentation works and that we can obtain traces, on each version:
    • create an instrumented executable.
    • run mts creating a trace script using that executable
    • run the trace script created:
      • verify that it executes just like a runall script in all respects, with either output-to-screen and output-redirected
      • verify that after execution there are traces, one per test, in your traces directory. Do ls -l on these and verify that no trace files are size "0".
      • run the th_builder on the trace dir, verify that it completes
      • run the th_printer on the th file created, verify that it completes
      • repeat this for both statement and branch traces
  4. To verify that instrumentation doesn't inadvertenly alter program behavior, on each version:
    • Repeat step 2, but this time, when you ask mts to make the runall-and-diff script, give it your instrumented executable name.
    • Have it do the diffs against the output directory you saved from the runs in step 2. Do this for both branch and statement traces.
    • Verify that the tests are creating the same outputs under the instrumented version that they did under the non-instrumented one.
  5. Generate a fault matrix for the object. Then generate it again and check that there are no differences; differences may indicate a source of non-determinism.