Running the MTS Tool

Make-test-script (mts) is the program that turns a universe file into a script to automatically execute test cases. The types of scripts that can be created by mts, and their respective uses, are described in detail in the script types section.

To run mts, type:

mts <object-dir> <object-exe> <universe-file> <script-type>
            <script-name> <program-suffix> <compare-dir>

(The carriage return above is for readability only; all parameters should be given on a single command line.)

<object-dir> : name of object directory for program under test
<object-exe> : name of executable to be executed,
<universe-file> : name of universe file
<script-type> : R|D|d|T|d_t|D_t
For runall, runall-and-diff using cmp, runall-and-diff using diff, trace, runall-and-diff-time script using diff, or runall-and-diff-time script using cmp
<nameScript> : name of output script
<program-suffix> : suffix of program name, or NULL (case insensitive)
-- only for script type T.
<compare-dir> : name of directory containing outputs to compare test outputs to, or NULL (case insensitive)
--only for script type D

It's important to understand how mts uses the first two parameters in creating a script. The executable name is the name of the executable that will be run, you could give its full path name, or give a path relative to where you'll be running the script from (the scripts directory).

The contents of the first parameter are more complex: they are prepended to names of input files, output files, scripts, and files-to-be-copied, allowing the script to put things in or get them from the right place. What you put here depends on where you run the script from. Examples will illustrate.

  1. Example:

    Suppose your universe file contains:

    -P [1 3 string1] -I [infile] -F [lex.yy.c|test1out]
-P [4 0 string2] -O [odir/out7.dat]

    Suppose you have the object "foo" present under the directory structure "/nfs/phantom/u8/goel2/objects/foo" and suppose you'll be running the "mts" program from the scripts directory, i.e at path "/nfs/phantom/u8/goel2/objects/foo/scripts".

    You type:

    mts .. ../source/foo.exe ../testplans.alt/universe sc.sh R NULL NULL

    This causes mts to look for universe file "universe" at path "../testplans.alt/universe" relative to the script directory path which is at "/nfs/phantom/u8/goel2/objects/foo/scripts", so mts finds the universe file at "/nfs/phantom/u8/goel2/objects/foo/testplans.alt/universe, and writes a runall script sc.sh looking like this:

    echo script type: R
echo ">>>>>>>>running test 1"
../source/foo.exe  1 3 string1 < ../inputs/infile > ../outputs/t1
mv ../scripts/lex.yy.c ../outputs/test1.out
echo ">>>>>>>>running test 2"
../source/foo.exe  4 0 string2 > ../outputs/odir/out7.dat

    NOTES: mts has taken your first parameter, "..", and used it in writing where infile and outputs are (e.g. writing "../inputs/infile" on test 1)

  2. Example:

    Suppose you have the same universe file, and you type:

    mts /nfs/phantom/u8/goel2/objects/foo
  /nfs/phantom/u8/goel2/objects/foo/source/foo.exe
  ../testplans.alt/universe sc.sh R NULL NULL

    This causes mts to look for universe file "universe", and write a runall script looking like this:

    echo script type: R
echo ">>>>>>>>running test 1"
/nfs/phantom/u8/goel2/objects/foo/source/foo.exe  1 3 string1 <
/nfs/phantom/u8/goel2/objects/foo/inputs/infile >
/nfs/phantom/u8/goel2/objects/foo/outputs/t1
mv /nfs/phantom/u8/goel2/objects/foo/scripts/lex.yy.c /nfs/phantom/u8/goel2/objects/foo/outputs/test1.out
echo ">>>>>>>>running test 2"
/nfs/phantom/u8/goel2/objects/foo/source/foo.exe  4 0 string2 >
/nfs/phantom/u8/goel2/objects/foo/outputs/odir/out7.dat

Types of test scripts

There are several types of scripts that can be created from a STImpL file; we describe each here.

  1. "Runall" script.

    The purpose of a "runall" script is to execute every test listed in a STImpL file, saving outputs in the outputs directory. To illustrate the possibilities, we provide examples of the various parameters, with STImpL files and the scripts that get created from them if those parameters are used.

    Remember that the resulting script is expected to run from the "scripts" directory of the object program. Further, assume that the executable for the object program will be found in the "source" directory with the name foo.exe

    In the following, suppose the object program is "foo.c".

    1. -P parameter.

      Given STImpL file:

      -P [1 3 string1]
-P [4 0 string2]

      A resulting runall script is:

      The following is an R script type
echo ">>>>>>>>running test 1"
../source/foo.exe  1 3 string1 > ../outputs/t1
echo ">>>>>>>>running test 2"
../source/foo.exe  4 0 string2 > ../outputs/t2

      NOTES: since -O wasn't used, default output files are used;
      parameters are put in place after the executable name.

    2. -I parameter.

      Given STImpL file:

      -I [in17.dat]
-I [idir/in16.dat]

      A resulting runall script is:

      The following is an R script type
echo ">>>>>>>>running test 1"
../source/foo.exe  < ../inputs/in17.dat > ../outputs/t1
echo ">>>>>>>>running test 2"
../source/foo.exe  < ../inputs/idir/in16.dat > ../outputs/t2

      NOTES: since -O wasn't used, default output files are used;
      no parameters are specified in the STImpL file, so none are in the script;
      input is piped from input files in the inputs directory.

    3. -O parameter with -P.

      Given STImpL file:

      -P [1 3 string1] -O [out1.dat]
-P [4 0 string2] -O [odir/out7.dat]

      A resulting runall script is:

      The following is an R script type
echo ">>>>>>>>running test 1"
../source/foo.exe  1 3 string1 > ../outputs/out1.dat
echo ">>>>>>>>running test 2"
../source/foo.exe  4 0 string2 > ../outputs/odir/out7.dat

      NOTES: -O causes standard output to go where specified;
      parameters are put in place after the executable name.

    4. -F parameter with -P.

      Given STImpL file:

      -P [1 3 string1] -F [lex.yy.c|test1out]
-P [4 0 string2] -F [lex.yy.c|test2out]

      A resulting runall script is:

      The following is an R script type
echo ">>>>>>>>running test 1"
../source/foo.exe  1 3 string1 > ../outputs/t1
mv ../scripts/lex.yy.c ../outputs/test1out
echo ">>>>>>>>running test 2"
../source/foo.exe  4 0 string2 > ../outputs/t2
mv ../scripts/lex.yy.c ../outputs/test2out

      NOTES: since -O wasn't used, default output files are used;
      parameters are put in place after the executable name;
      stdout output is still put into output files;
      unix "mv" command does the requested file move.

    5. -s and -x parameters with (possibly)-P

      Given STImpL file:

      -P [-Dvarname] -s [Sscript1] -x [Xscript1]
-s [Sscript1] -x[Xscript1]

      And suppose that files Sscript1 and Xscript1 contain the following:

      Sscript1:
---------
cp -r ../inputs/setupdirs/sdir1 ./workingdir
cd workingdir

Xscript1:
---------
cd ..
rm -r ./workingdir

      A resulting runall script is:

      The following is an R script type
echo ">>>>>>>>running test 1"
cp -r ../inputs/setupdirs/sdir1 ./workingdir
cd workingdir
../source/foo.exe  -Dvarname > ../outputs/t1
cd ..
rm -r ./workingdir
echo ">>>>>>>>running test 2"
cp -r ../inputs/setupdirs/sdir1 ./workingdir
cd workingdir
../source/foo.exe  > ../outputs/t2
cd ..
rm -r ./workingdir

      NOTES: since -O wasn't used, default output files are used;
      parameters are put in place after the executable name on test 1;
      text of startup and exit script files has been placed into the script before and after program foo.exe gets executed.

    6. -S and -X parameters

      Given STImpL file:

      -S [script1 p1] -X[script2]

      A resulting runall script is:

      The following is an R script type
echo ">>>>>>>>running test 1"
../testplans.alt/testscripts/script1 p1
../source/foo.exe  > ../outputs/t2
../testplans.alt/testscripts/script2
  2. "Get-traces" script.

    The get-traces script is almost exactly like a runall script. The difference is that it is intended to run an instrumented executable, and after each run, save the trace file created for that executable.

    (These scripts depend on availability of Aristotle. It should be possible, however, to modify the mts code to cause it to create scripts utilizing some other coverage tool.)

    Because of the similarity of this script with the runall script, we give just one example, with the -P parameter. We assume that the instrumented executable is "foo.int.exe" in the "objects/foo/source" subdirectory.

    Given STImpL file:

    -P [1 3 string1]
-P [4 0 string2]

    A resulting get-traces script is:

    The following is an T script type
echo ">>>>>>>>running test 1"
../source/foo.int.exe  1 3 string1 > ../outputs/t1
cp $ARISTOTLE_DB_DIR/foo.c.tr ../traces/0.tr
echo ">>>>>>>>running test 2"
../source/foo.int.exe  4 0 string2 > ../outputs/t2
cp $ARISTOTLE_DB_DIR/foo.c.tr ../traces/1.tr

    NOTES: since -O wasn't used, default output files are used;
    parameters are put in place after the executable name;
    the executable is foo.int.exe.
    We have an extra line after each test copying the trace file to the traces subdirectory.
    Trace files are numbered from 0 up; test case k generates trace file (k-1).tr (unfortunate but can't be helped)

  3. "Runall-and-Diff" script.

    The runall-and-diff script is also quite like a runall script. The difference is that after running each test, it compares results of that test with results from a previous run. As such, the script is automating regression testing.

    What the script has to do is compare anything created as output by the program with saved versions of outputs. That means it must accomodate 1) files written to standard-output, 2) output files created by the program. It detects the need to do comparisons with these, and the names of files to compare with, by looking in -O and -F parameters.

    Because of the similarity of this script with the runall script, we give just one example, with the -P parameter, -O, and -F used. We assume that the executable is "foo.exe" in the"objects/foo/source" subdirectory, and that the outputs directory is "/nfs/spectre/u5/mydir/oldoutputs/".

    Given STImpL file:

    -P [1 3 string1] -F [lex.yy.c|test1out]
-P [4 0 string2] -O [odir/out7.dat] -F [lex.yy.c|test2out]

    A resulting runall-and-diff script is:

    The following is an D script type
echo ">>>>>>>>running test 1"
../source/foo.exe  1 3 string1 > ../outputs/t1
mv ../scripts/lex.yy.c ../outputs/test1out
cmp -s ../outputs/t1 /nfs/spectre/u5/mydir/oldoutputs/t1
cmp -s ../outputs/test1out /nfs/spectre/u5/mydir/oldoutputs/test1out

echo ">>>>>>>>running test 2"
../source/foo.exe  4 0 string2 > ../outputs/odir/out7.dat
mv ../scripts/lex.yy.c ../outputs/test2out
cmp -s ../outputs/out7.dat /nfs/spectre/u5/mydir/oldoutputs/out7.dat
cmp -s ../outputs/test2out /nfs/spectre/u5/mydir/oldoutputs/test2out

    NOTES: -O causes standard output to go where specified;
    parameters are put in place after the executable name;
    the output file specifed with -O in the second test gets compared with a previous one.
    On the first test, where no -O appears, the "default" output file t1 gets compared with a previous one.
    The output files listed in -F get compared with previous ones;
    the location of old outputs in this example, /nfs/spectre/u5/mydir/oldoutputs/, is something that can be specified when mts is run to make the script, but it is assumed that that directory is a saved copy of an outputs directory created on some run of another script, and that the directory has the same structure and filenames in it as will be created by this run of this script.

    If you run mts with "-d" rather than "-D", the program uses "diff -r" instead of "cmp -s" to do the differencing. This allows comparison of entire output directories.

  4. "Runall-and-Diff-Time" script.

    The runall-and-diff time script is almost exactly like a runall script. The difference is that it is intended to collect the time taken by each test to run.

    Because of the similarity of this script with the runall-and-diff script, we give just one example, with the -P parameter, -O, and -F used. We assume that the instrumented executable is "foo.exe" in the "objects/foo/source" subdirectory, and that the outputs directory is "/nfs/spectre/u5/mydir/oldoutputs/".

    Given STImpL file:

    -P [1 3 string1] -F [lex.yy.c|test1out]
-P [4 0 string2] -O [odir/out7.dat] -F [lex.yy.c|test2out]

    A resulting runall-and-diff time script is:

    The following is an D_t script type
echo ">>>>>>>>running test 1"
set t=`mydate`
echo "Start time = ${t}"
../source/foo.exe  1 3 string1 > ../outputs/t1
mv ../scripts/lex.yy.c ../outputs/test1out
cmp -s ../outputs/t1 /nfs/spectre/u5/mydir/oldoutputs/t1
cmp -s ../outputs/test1out /nfs/spectre/u5/mydir/oldoutputs/test1out
set t=`mydate`
echo "Finish time = ${t}"

echo ">>>>>>>>running test 2"
set t=`mydate`
echo "Start time = ${t}"
../source/foo.exe  4 0 string2 > ../outputs/odir/out7.dat
mv ../scripts/lex.yy.c ../outputs/test2out
cmp -s ../outputs/out7.dat /nfs/spectre/u5/mydir/oldoutputs/out7.dat
cmp -s ../outputs/test2out /nfs/spectre/u5/mydir/oldoutputs/test2out
set t=`mydate`
echo "Finish time = ${t}"

    Here mydate is the c program which returns sytem time in seconds (elapsed from 1970) and in fractions of a second with resolution of 1 microsecond.

    If you run mts with "-d_t" rather than "-D_t", the program uses "diff -r" instead of "cmp -s" to do the differencing.