STImpL File Specification

A STImpL file is a file listing your test cases.

We've created a program, make-test-script (see mts for C, javamts for Java), that takes a STImpL file (along with other parameters) and creates a script that automatically executes the tests listed in that file.

Note that we implement a "test suite" as a subset (not necessarily proper) of the lines in a STImpL file. So anything that works on a STImpL file should work on a test suite.

This document contains specifications for STImpL files and for the scripts that mts/javamts creates.

Format of a STImpL File

Each line (string of characters terminated by a newline) in a STImpL file represents either a test case or an assignment of a value to a variable. STImpL uses C-shell syntax to assign a value to variable. For example, you could use the following to set or change the value of the CLASSPATH environment variable:

setenv CLASSPATH <classpath>

You may notice that some existing STImpL files use Bourne shell syntax to set the CLASSPATH variable. This is a special case that is only supported for the CLASSPATH variable, and should not be used in future STImpL files.

All other lines in a STImpL file specify test cases, constructed from combinations of the following parameters.

Ideally each test case should be unique, but some earlier objects' STImpL files are not completely unique.

  1. Lines in the file contain one or more parameters; valid parameters and their meaning are as follows (text enclosed in <> stands for values that get put in, all other characters are literal):

    -P [<parameter-string>]

    For use in supplying parameters.

    <parameter-string> represents the string of characters you would type as parameters to the object program for that test case. These will not be "interpreted" in any way by mts, that program will simply copy this string verbatim and use it in generating a test script.

    For program runs (test cases) that require no parameters, the -P option won't be used in the STImpL file line for that test.

    -I [<input-file-name>]

    For use in supplying standard input only.

    <input-file-name> represents the name of a file that contains input to the program under test, that will be redirected to the program-under-test using the unix "<" redirection mechanism

    This file name, however, must be a filename relative to the "inputs" subdirectory of the object program. I.e. if your input file is "foo1.dat" and resides directly in the "inputs" subdirectory for "foo" you would write "-I [foo1.dat]. If your input file is "foo1.dat" but you've placed it in a subdirectory of the inputs directory entitled "stdinputs", you would write "-I [stdinputs/foo1.dat].

    For program runs (test cases) that require no standard input, the -I option won't be used in the STImpL file line for that test. There is no default input file.

    Note that <input-file-name> must be a string in legal unix filename syntax. In particular it can't contain any ">" characters.

    -O [<output-file-name]

    For use in redirecting standard-output only.

    <output-file-name> represents the name of a file that will hold standard output of the program under test, that output will be redirected to this file using the unix ">" redirection mechanism

    This file name, however, must be a filename relative to the "outputs" subdirectory of the object program. For example if your output file is to be "foo1.out" and is to be placed directly in the "outputs" subdirectory for "foo" you would write "-O [foo1.out]". If your output file is "foo1.out" but you want it placed in a subdirectory of the outputs directory entitled "stdoutputs", you would write "-I [stdoutputs/foo1.out].

    When the -O option is omitted, the test script will by default pipe all standard output to the outputs directory, creating a file named tk, where k is an integer representing the test number of this test (the number of the line in the STImpL file that this test case was taken from).

    Note that <output-file-name> must be a string in legal unix filename syntax. In particular it can't contain any ">" characters.

    -F [<name-of-file-to-move>|<name-of-new-file>]

    For use in causing output files created by the program to be moved to the official outputs area.

    The -O parameter provides a way to specify where standard output should go, and when output files are specified as parameters they are handled in -P. But sometimes a program creates a file with a default name not specified as a parameter. We need to be able to get this file to the outputs directory, potentially under a different name. The -F parameter allows this.

    The parameter takes two arguments inside the brackets, separated by the "|" character. The <name-of-file-to-move> argument names the file created by the program under test. The <name-of-new-file> names where you want this file moved.

    The <name-of-new-file> file name must be a filename relative to the "outputs" subdirectory of the object program. The <name-of-file-to-move> file name must be a filename relative to the "scripts" subdirectory of the object program, where the test is being run from.

    For example if you want your output file to be "foo1.out" and to be placed directly in the "outputs" subdirectory for "foo" but your program will be putting it in the current directory (scripts) and calling it lex.yy.c, you would write "-F[lex.yy.c|foo1.out].

    At most three -F parameters can occur in a STImpL line, and each will be translated into code to move the named file.

    Finally, since this parameter activates a "mv" command, the "file" to be moved could just as well be a directory.

    -S [<name-of-startup-script> <params>]
    -X [<name-of-exit-script> <params>]

    If the program under test has special requirements it may be impossible to capture them with generic options such as those above. This pair of parameters will allow the tester to create pieces of script that should be executed before and/or after the program-under-test is run.

    Either or both of these parameters can be present.

    Typically, the <name-of-startup-script> and <name-of-exit-script> arguments each name a file that contains some lines of C shell script. Such filenames must be filenames relative to the "testplans.alt/testscripts" subdirectory of the object program. For example if your script file is to be "test1startup.sh" then it has to be placed directly in the "testplans.alt/testscripts" subdirectory for "foo" and you would write "-S [test1startup.sh]. If your script file is "test1startup.sh" but you want it placed in a subdirectory of "testplans.alt/testscripts" entitled "scripts", you would write "-S [scripts/test1startup.sh]."

    If your script needs some parameters of some sort, you can supply them as shown above by <params>.

    If you use -S, mts inserts a line to execute the script you name, just prior to the line that executes the program under test. If you use -X, mts inserts a line to execute the script you name, just after the line that executes the program under test. So the script written by mts ends up invoking your scripts directly. You might need to know that in this case, these scripts are invoked as sub-shells, and on termination the environment is restored to its previous state. If that's not what you need, see "-s" and "-x" below.

    (Note: there's nothing stopping <name-of-startup-script> and <name-of-exit-script> arguments from naming binaries or executables of other, non-script, sorts, and the <params> as being parameters in those, if those binaries are executables and are in the proper testplans subdirectory.)

    -s [<name-of-startup-script>]
    -x [<name-of-exit-script>]

    These are like -S and -X, except faced with them, mts physically copies the lines from the named scripts into the script it creates. This may be needed if you need to run some lines that can't be run from a subshell. In this case, you can't use parameters.

    There's nothing stopping you from using both -S and -s, and/or both -X and -x, if you use both -S and -s they'll be processed (and result in lines in the script) in order -S, -s and if you use both -X and -x they'll be processed (and result in lines in the script) in order -x -X. (In other words, the -s and -x calls are nested within the -S and -X calls.)

    -D [<test-driver-name>]

    For use in specifying a test driver.

    <test-driver-name> represents the name of an executable to be executed. If your script needs different executables for different test cases, then you can supply these as shown above by <test-driver-name>.

    For a test script that requires only one executable file, the -D option won't be used in the STImpL file. Instead, an executable file name is given as a parameter to javamts.

    -B

    For use in causing executable program to be run as a background job.

    -C [comment]

    When present in a STImpL file spec, causes any text appearing inside the brackets to be ignored by mts. Comments are used in a STImpL file to specify which STImpL line corresponds to which test frame.

  2. On each STImpL file line, at least one parameter must be given. Parameters may be listed in any order, all must begin with hyphen followed by a character with no spaces in between them. Zero or more spaces (blanks, not tabs) may appear after the hyphen/character pair, brackets must enclose and delimit the text between them, that text can include anything except for a newline or bracket. Parameters may be separated by zero or more spaces (blanks, not tabs), an end-of-line is interpreted to end the set of parameters, and the entire line cannot exceed 4096 characters (adjustable by changing a constant). Parameters are case-sensitive (-I doesn't mean the same thing as -i).
  3. Instructions for naming input and output files in parameter strings.
    1. input files

      Assume that all tests will be run from the "scripts" subdirectory of the object; i.e. if your object is "calc", then tests will be run from "calc/scripts". We'll generate our automated testing scripts in that directory and that's where we'll run those scripts from.

      So, if you want your input files to come from the inputs directory, and need to specify them with parameters, you'd have to specify the path relative to the scripts directory in the parameter list.

      Example:

      Suppose your program foo opens and reads an input file, and this file is specified in a parameter -i<inputfile>, and suppose for one of the tests of foo you want it to use a file "myinput1.dat".

      Suppose you put "myinput1.dat" in a subdirectory of "foo/inputs" called "inputfiles".

      Now, if you were running this program by hand on this test from the scripts directory you'd type:

      ../source/foo.exe -i../inputs/inputfiles/myinput1.dat

      So, in your STImpL file spec for this you'd write:

      -P [-i../inputs/inputfiles/myinput1.dat]
    2. output files

      There are two ways to get output files into the output directory when specifying them with parameters.

      However, only use of the first will support the use, later, of "Runall-and-Diff" (regression testing) scripts.

      1. Write the file to the local directory and include use of the -F option to cause it to be moved.

        So considering the example from C.2.b., you could write:

        -P [-omyoutput1.dat] -F[myoutput1.dat|outputfiles/myoutput1.dat]
      2. You could do as with input files, assuming that the program will be run from "scripts", and that the parameter should be rewritten to redirect the output up to ../outputs/outputfilename

        Example:

        Suppose your program foo opens and writes to an output file, and this file is specified in a parameter -o<outputfile>, and suppose for one of the tests of foo you want it to use a file "myoutput1.dat".

        Suppose you want to put "myoutput1.dat" in a subdirectory of "foo/outputs" called "outputfiles".

        Now, if you were running this program by hand on this test from the scripts directory you'd type:

        ../source/foo.exe -o../outputs/outputfiles/myoutput1.dat

        So, in your STImpL file spec for this you'd write:

        -P [-o../outputs/outputfiles/myoutput1.dat]

        But NOTE, as mentioned above, we can't write Runall-and-Diff scripts easily that will use output files written in this manner, so unless there's a good reason, you shouldn't use this approach, and we mention it here only for historical reasons.