Running the JavaMTS Tool

Java make-test-script (javamts) is the program that turns a universe file into a script to automatically execute test cases. The types of scripts that can be created by javamts, and their respective uses, are described in detail in the in the script types section.

To run javamts, type:

javamts  <object-dir> <object-exe> <universe-file> <script-type>
            <script-name> <program-suffix> <compare-dir>  <instrument-option>

(The carriage return above is for readability only; all parameters should be given on a single command line.)

<object-dir> : name of object directory for program under test
<object-exe> : name of executable to be executed,
<universe-file> : name of universe file
<script-type> : R|D|d|T|d_t|D_t
For runall, runall-and-diff using cmp, runall-and-diff using diff, trace, runall-and-diff-time script using diff, or runall-and-diff-time script using cmp
<nameScript> : name of output script
<program-suffix> : suffix of program name, or NULL (case insensitive)
-- only for script type T.
<compare-dir> : name of directory containing outputs to compare test outputs to, or NULL (case insensitive)
--only for script type D
<instrument-option> : -B|E|X|C, or NULL (case insensitive)
--only for script type T

It's important to understand how javamts uses the first two parameters in creating a script. The executable name is the name of the executable that will be run, you could give its full path name, or give a path relative to where you'll be running the script from (the scripts directory).

The contents of the first parameter are more complex: they are prepended to names of input files, output files, scripts, and files-to-be-copied, allowing the script to put things in or get them from the right place. What you put here depends on where you run the script from. Examples will illustrate.

Suppose you have the object "foo" present under the directory structure "/nfs/phantom/u8/goel2/objects/foo" and suppose you'll be running the javamts program from the scripts directory, i.e at path "/nfs/phantom/u8/goel2/objects/foo/scripts".

  1. Example:

    Suppose your universe file contains:

    setenv CLASSPATH ../source
-P [1 3 string1]  -I [infile] -F [lex.yy.c|test1out]
-P [4 0 string2] -O [odir/out7.dat]

    You type:

    javamts .. foo ../testplans.alt/universe
    R sc.sh NULL NULL NULL

    This causes javamts to look for STImpL file "universe" at path "../testplans.alt/universe" relative to the script directory path which is at "/nfs/phantom/u8/goel2/objects/foo/scripts", so javamts finds the STImpL file at "/nfs/phantom/u8/goel2/objects/foo/testplans.alt/universe, and writes a runall script sc.sh looking like this:

    echo script type: R
unsetenv CLASSPATH
setenv CLASSPATH ../source
echo ">>>>>>>>running test 1"
java foo 1 3 string1 < ../inputs/infile  >& ../outputs/t1
mv ../scripts/lex.yy.c ../outputs/test1.out
echo ">>>>>>>>running test 2"
java foo 4 0 string2  >& ../outputs/odir/out7.dat

    NOTES: javamts has taken your first parameter, "..", and used it in writing where infile and outputs are (e.g. writing "../inputs/infile" on test 1)

  2. Example:

    Suppose you have the same STImpL file, and you type:

    javamts /nfs/phantom/u8/goel2/objects/foo
java foo ../testplans.alt/universe R sc.sh NULL NULL NULL

    This causes javamts to look for STImpL file "universe", and write a runall script looking like this:

    echo script type: R
unsetenv CLASSPATH
setenv CLASSPATH ../source
echo ">>>>>>>>running test 1"
java foo  1 3 string1 < /nfs/phantom/u8/goel2/objects/foo/inputs/infile >& /nfs/phantom/u8/goel2/objects/foo/outputs/t1
mv /nfs/phantom/u8/goel2/objects/foo/scripts/lex.yy.c /nfs/phantom/u8/goel2/objects/foo/outputs/test1.out
echo ">>>>>>>>running test 2"
java foo  4 0 string2 >&
/nfs/phantom/u8/goel2/objects/foo/outputs/odir/out7.dat
  3. Example:

    Suppose your STImpL file contains:

    setenv CLASSPATH ../source
-D [fooTest] -P [1 3 string1]  -I [infile] -F [lex.yy.c|test1out]
-D [fooTest1] -P [4 0 string2] -O [odir/out7.dat]

    You type:

    javamts .. null ../testplans.alt/universe  
    R sc.sh NULL NULL NULL

    This causes javamts to look for STImpL file "universe" at path "../testplans.alt/universe" relative to the script directory path which is at "/nfs/phantom/u8/goel2/objects/foo/scripts", so javamts finds the STImpL file at "/nfs/phantom/u8/goel2/objects/foo/testplans.alt/universe, and writes a runall script sc.sh looking like this:

    echo script type: R
unsetenv CLASSPATH
setenv CLASSPATH ../source
echo ">>>>>>>>running test 1"
java fooTest 1 3 string1 < ../inputs/infile  >& ../outputs/t1
mv ../scripts/lex.yy.c ../outputs/test1.out
echo ">>>>>>>>running test 2"
java fooTest1 4 0 string2  >& ../outputs/odir/out7.dat

    NOTES: javamts has taken your first parameter, "..", and used it in writing where infile and outputs are (e.g. writing "../inputs/infile" on test 1). Since the STImpL file has -D option, the second parameter of javamts is ignored (here, null is given, but any name can be provided).

  4. Example:

    Suppose your STImpL file contains:

    setenv CLASSPATH ../source:<BCEL>:<sofya>
-D [foo_1] -P [1] -I [infile]
-D [foo_2] -P [4 0] -O [odir/out.dat]

    You type:

    javamts .. sofya.ed.BBTracer ../testplans/universe
    T sc.sh instout.tr NULL -B

    This causes javamts to look for STImpL file "universe" at path "../testplans.alt/universe" relative to the script directory path which is at "/nfs/phantom/u8/goel2/objects/foo/scripts", so javamts finds the STImpL file at "/nfs/phantom/u8/goel2/objects/foo/testplans.alt/universe, and writes a gettraces script sc.sh looking like this:

    echo script type: T
unsetenv CLASSPATH
setenv CLASSPATH ../source:<BCEL>:<sofya>
echo ">>>>>>>>running test 1"
java sofya.ed.BBTracer -B foo_1 1 < ../inputs/infile >& ../outputs/t1
cp ${HOME}/.sofyadb/instout.tr ../traces/0.tr
echo ">>>>>>>>running test 2"
java sofya.ed.BBTracer -B foo_2 4 0 >& ../outputs/odir/out7.dat
cp ${HOME}/.sofyadb/instout.tr ../traces/1.tr

    NOTES: <BCEL> is the full path to the location of the BCEL library;
    <sofya> is the full path to the Sofya system directory.

  5. Example:

    Suppose your STImpL file contains:

    setenv CLASSPATH ../source/src:../source/junit
-P[de.susebox.java.util.TestTokenizerProperties]
-P[de.susebox.java.util.TestTokenProperties]

    You type:

    javamts .. junit.textui.TestRunner
    ../testplans.alt/v0/universe.junit R sc.sh NULL NULL NULL

    This causes javamts to look for STImpL file "universe.junit" at path "../testplans.alt/v0/universe.junit" relative to the script directory path which is at "/nfs/phantom/u8/goel2/objects/foo/scripts", so javamts finds the STImpL file at "/nfs/phantom/u8/goel2/objects/foo/testplans.alt/v0/universe.junit", and writes a runall script sc.sh looking like this:

    echo script type: R
unsetenv CLASSPATH
setenv CLASSPATH ../source/src:../source/junit
echo ">>>>>>>>running test 1"
java junit.textui.TestRunner de.susebox.java.util.TestTokenizerProperties  > ../outputs/t1 2>&1
echo ">>>>>>>>running test 2"
java junit.textui.TestRunner de.susebox.java.util.TestTokenProperties  > ../outputs/t2 2>&1

Types of test scripts

There are several types of scripts that can be created from a STImpL file; we describe each here.

  1. "Runall" script.

    The purpose of a "runall" script is to execute every test listed in a STImpL file, saving outputs in the outputs directory. To illustrate the possibilities, we provide examples of the various parameters, with STImpL files and the scripts that get created from them if those parameters are used.

    Remember that the resulting script is expected to run from the "scripts" directory of the object program. Further, assume that the executable for the object program will be found in the "source" directory with the name foo.

    In the following, suppose the object program is "foo.java".

    1. -D parameter.

      For this option, we assume that the executables for the object programs (actually, test driver names) are fooTest.class and fooTest1.class instead of foo.class.

      Given STImpL file:

      setenv CLASSPATH ../source
-D [fooTest]
-D [fooTest1]

      A resulting runall script is:

      The following is an R script type
unsetenv CLASSPATH
setenv CLASSPATH ../source
echo ">>>>>>>>running test 1"
java fooTest >& ../outputs/t1
echo ">>>>>>>>running test 2"
java fooTest1 >& ../outputs/t2

      NOTES: since -O wasn't used, default output files are used;
      parameters are put in place after the executable name

    2. -P parameter.

      Given STImpL file:

      setenv CLASSPATH ../source
-P [1 3 string1]
-P [4 0 string2]

      A resulting runall script is:

      The following is an R script type
unsetenv CLASSPATH
setenv CLASSPATH ../source
echo ">>>>>>>>running test 1"
java foo 1 3 string1  >& ../outputs/t1
echo ">>>>>>>>running test 2"
java foo 4 0 string2  >& ../outputs/t2

      NOTES: since -O wasn't used, default output files are used;
      parameters are put in place after the executable name

    3. -I parameter.

      Given STImpL file:

      setenv CLASSPATH ../source
-I [in17.dat]
-I [idir/in16.dat]

      A resulting runall script is:

      The following is an R script type
unsetenv CLASSPATH
setenv CLASSPATH ../source
echo ">>>>>>>>running test 1"
java foo < ../inputs/in17.dat  >& ../outputs/t1
echo ">>>>>>>>running test 2"
java foo < ../inputs/idir/in16.dat  >& ../outputs/t2

      NOTES: since -O wasn't used, default output files are used;
      no parameters are specified in the STImpL file, so none are present in the script;
      input is piped from the input file in the inputs directory

    4. -O parameter with -P.

      Given STImpL file:

      setenv CLASSPATH ../source
-P [1 3 string1] -O [out1.dat]
-P [4 0 string2] -O [odir/out7.dat]

      A resulting runall script is:

      The following is an R script type
unsetenv CLASSPATH
setenv CLASSPATH ../source
echo ">>>>>>>>running test 1"
java foo 1 3 string1  >& ../outputs/out1.dat
echo ">>>>>>>>running test 2"
java foo 4 0 string2  >& ../outputs/odir/out7.dat

      NOTES: -O causes standard output to go where specified;
      parameters are put in place after the executable name

    5. -F parameter with -P.

      Given STImpL file:

      setenv CLASSPATH ../source
-P [1 3 string1] -F [lex.yy.c|test1out]
-P [4 0 string2] -F [lex.yy.c|test2out]

      A resulting runall script is:

      unsetenv CLASSPATH
setenv CLASSPATH ../source
echo ">>>>>>>>running test 1"
java foo 1 3 string1  >& ../outputs/t1
mv ../scripts/lex.yy.c ../outputs/test1out
echo ">>>>>>>>running test 2"
java foo 4 0 string2  >& ../outputs/t2
mv ../scripts/lex.yy.c ../outputs/test2out

      NOTES: since -O wasn't used, default output files are used;
      parameters are put in place after the executable name;
      stdout output is still put into output files;
      unix mv command does the requested file move.

    6. -s and -x parameters with (possibly) -P

      Given STImpL file:

      setenv CLASSPATH ../source
-P [-Dvarname] -s [Sscript1] -x [Xscript1]
-s [Sscript1] -x[Xscript1]

      And suppose that files Sscript1 and Xscript1 contain the following:

      Sscript1:
---------
cp -r ../inputs/setupdirs/sdir1 ./workingdir
cd workingdir

Xscript1:
---------
cd ..
rm -r ./workingdir

      A resulting runall script is:

      The following is an R script type
unsetenv CLASSPATH
setenv CLASSPATH ../source
echo ">>>>>>>>running test 1"
cp -r ../inputs/setupdirs/sdir1 ./workingdir
cd workingdir
java foo -Dvarname  >& ../outputs/t1
cd ..
rm -r ./workingdir
echo ">>>>>>>>running test 2"
cp -r ../inputs/setupdirs/sdir1 ./workingdir
cd workingdir
java foo >& ../outputs/t2
cd ..
rm -r ./workingdir

      NOTES: since -O wasn't used, default output files are used;
      parameters are put in place after the executable name on test 1;
      text of startup and exit script files has been placed into the script before and after program foo.class gets executed.

    7. -S and -X parameters

      Given STImpL file:

      setenv CLASSPATH ../source
-S [script1 p1] -X[script2]

      A resulting runall script is:

      The following is an R script type
unsetenv CLASSPATH
setenv CLASSPATH ../source
echo ">>>>>>>>running test 1"
../testplans.alt/testscripts/script1 p1
java foo >& ../outputs/t2
../testplans.alt/testscripts/script2
    8. -B parameter
      1. For C shell

        Given STImpL file:

        setenv CLASSPATH ../source
-P [1 3 string1] -O [out1.dat] -B

        A resulting runall script is:

        The following is an R script type
unsetenv CLASSPATH
setenv CLASSPATH ../source
echo ">>>>>>>>running test 1"
java foo 1 3 string1  >& ../outputs/out1.dat &
      2. For Bourne shell

        Given STImpL file:

        CLASSPATH=../source
-P [1 3 string1] -O [out1.dat] -B

        A resulting runall script is:

        The following is an R script type
unset CLASSPATH
CLASSPATH=../source
export CLASSPATH
echo ">>>>>>>>running test 1"
java foo 1 3 string1  >& ../outputs/out1.dat 2>&1 &
  2. "Get-traces" script.

    The get-traces script is almost exactly like a runall script. The difference is that it is intended to run an instrumented executable, and after each run, save the trace file created for that executable.

    (The get-traces script is set up to run the Sofya tracer; but you could substitute another trace gathering utility, and modify javamts to use it.)

    The instrumented executable is invoked by sofya.ed.BBTracer, which gathers trace files for that executable. The instrumented executable is given with -D option, which is for supplying a test driver name. We assume that the instrumented executable is "foo.class" in the "objects/foo/source" subdirectory.

    Given STImpL file:

    setenv CLASSPATH ../source:<BCEL>:<sofya>
-D [foo] -P [1 3 string1]

    You type: javamts .. sofya.ed.BBTracer ../testplans/STImpL T sc.sh instout.tr NULL -B

    A resulting get-traces script is:

    The following is an T script type
unsetenv CLASSPATH
setenv CLASSPATH ../source:<BCEL>:<sofya>
echo ">>>>>>>>running test 1"
java sofya.ed.BBTracer -B foo 1 3 string1  >& ../outputs/t1
cp ${HOME}/.sofyadb/instout.tr ../traces/0.tr

    NOTES: <BCEL> is the full path to the location of the BCEL library;
    <sofya> is the full path to the Sofya system directory.

  3. "Runall-and-Diff" script.

    The runall-and-diff script is also quite like a runall script. The difference is that after running each test, it compares results of that test with results from a previous run. As such, the script is automating regression testing.

    What the script has to do is compare anything created as output by the program with saved versions of outputs. That means it must accomodate 1) files written to standard-output, 2) output files created by the program. It detects the need to do comparisons with these, and the names of files to compare with, by looking in -O and -F parameters.

    Because of the similarity of this script with the runall script, we give just one example, with the -P parameter, -O, and -F used. We assume that the executable is "foo.class" in the"objects/foo/source" subdirectory, and that the outputs directory is "/nfs/spectre/u5/mydir/oldoutputs/".

    Given STImpL file:

    setenv CLASSPATH ../source
-P [1 3 string1]  -F [lex.yy.c|test1out]
-P [4 0 string2] -O [odir/out7.dat] -F [lex.yy.c|test2out]

    A resulting runall-and-diff script is:

    The following is an D script type
unsetenv CLASSPATH
setenv CLASSPATH ../source
echo ">>>>>>>>running test 1"
java foo 1 3 string1  >& ../outputs/t1
mv ../scripts/lex.yy.c ../outputs/test1out
cmp -s ../outputs/t1 /nfs/spectre/u5/mydir/oldoutputs/t1
cmp -s ../outputs/test1out /nfs/spectre/u5/mydir/oldoutputs/test1out

echo ">>>>>>>>running test 2"
java foo 4 0 string2  >& ../outputs/odir/out7.dat
mv ../scripts/lex.yy.c ../outputs/test2out
cmp -s ../outputs/out7.dat /nfs/spectre/u5/mydir/oldoutputs/out7.dat
cmp -s ../outputs/test2out /nfs/spectre/u5/mydir/oldoutputs/test2out

    NOTES: -O causes standard output to go where specified;
    parameters are put in place after the executable name;
    the output file specified with -O in the second test gets compared with a previous one.
    On the first test, where no -O appears, the "default" output file t1 gets compared with a previous one.
    The output files listed in -F get compared with previous ones.
    The location of old outputs in this example, /nfs/spectre/u5/mydir/oldoutputs/, is something that can be specified when javamts is run to make the script, but it is assumed that that directory is a saved copy of an outputs directory created on some run of another script, and that the directory has the same structure and filenames in it as will be created by this run of this script.

    If you run javamts with "-d" rather than "-D", the program uses "diff -r" instead of "cmp -s" to do the differencing. This allows comparison of entire output directories.

  4. "Runall-and-Diff-Time" script

    The runall-and-diff time script is almost exactly like a runall script. The difference is that it is intended to collect the time taken by each test to run.

    Because of the similarity of this script with the runall-and-diff script, we give just one example, with the -P parameter, -O, and -F used. We assume that the instrumented executable is "foo.class" in the "objects/foo/source" subdirectory, and that the outputs directory is "/nfs/spectre/u5/mydir/oldoutputs/".

    Given STImpL file:

    setenv CLASSPATH ../source
-P [1 3 string1]  -F [lex.yy.c|test1out]
-P [4 0 string2] -O [odir/out7.dat] -F [lex.yy.c|test2out]

    A resulting runall-and-diff time script is:

    The following is an D_t script type
unsetenv CLASSPATH
setenv CLASSPATH ../source
echo ">>>>>>>>running test 1"
set t=`mydate`
echo "Start time = ${t}"
java foo 1 3 string1  >& ../outputs/t1
mv ../scripts/lex.yy.c ../outputs/test1out
cmp -s ../outputs/t1 /nfs/spectre/u5/mydir/oldoutputs/t1
cmp -s ../outputs/test1out /nfs/spectre/u5/mydir/oldoutputs/test1out
set t=`mydate`
echo "Finish time = ${t}"

echo ">>>>>>>>running test 2"
set t=`mydate`
echo "Start time = ${t}"
java foo 4 0 string2  >& ../outputs/odir/out7.dat
mv ../scripts/lex.yy.c ../outputs/test2out
cmp -s ../outputs/out7.dat /nfs/spectre/u5/mydir/oldoutputs/out7.dat
cmp -s ../outputs/test2out /nfs/spectre/u5/mydir/oldoutputs/test2out
set t=`mydate`
echo "Finish time = ${t}"

    Here mydate is the c program which returns sytem time in seconds (elapsed from 1970) and in fractions of a second with resolution of 1 microsecond.

    If you run javamts with "-d_t" rather than "-D_t", the program uses "diff -r" instead of "cmp -s" to do the differencing.