C Object Directory Specification

Each object we create has its own object directory, named to indicate the program involved.

An object directory is organized into subdirectories (which in turn may contain subdirectories) according to a specific format. Doing this lets us write generic tools for experimentation that know where to find things and know what they're named, and work across all of our objects.

This document describes the object directory organization in terms of its subdirectories, and what they contain.

There are two types of organization for objects. One is our "previous" organization which is used on the Siemens programs and space, and is actually for any objects where we have a single base version v0, and several different versions that are derivatives of v0. The second type of organization is the "new" organization which is used for objects where we have a sequence v0, v1, v2... vk of versions, where each vk+1 is a derivative of the previous version vk.

In general, any directory or subdirectory that contains files whose function may not be obvious, or that depart from naming conventions, should contain a CONTENTS file describing those files.

The subdirectories of an object are as follows:

  1. source

    An empty directory. During experiments, a version of a program that we're working with is placed in here, then deleted after we're finished with it.

  2. versions.alt

    Contains "variants" of the source code of the versions of the program. Variants are needed when, say, we have to modify the code to accomodate some tool.

    The basic variant, that any object will have, is contained in the subdirectory of versions.alt called:

    versions.orig

    This directory contains subdirectories

    v0, v1, v2, v3 ... vK

    where v0 is the earliest version,and the other vJ contain the next sequential versions in turn.

    Each vK area contains the .c file and .h files needed to build the version, plus Makefiles, and subdirectories of any required non-system libraries. (Note that historically all non-library code in the objects was coalesced into a single .c file. This was to facilitate the use of a particular program analysis toolset, but is not a requirement and may not be true of future objects).

    Each Makefile must have the following targets:

    • build - creates <program>.exe executable file.
    • build-st - creates <program>.st.int.exe which is a statement instrumented executable.
    • build-bt - creates <program>.bt.int.exe which is a branch instrumented executable.

    (The last two commands are targets that can be used to indicate that the object should be compiled with statement and branch instrumentation, respectively. Any instrumentation tool can be used; we provide these targets to encourage the use of a common interface that scripts and other experiment tools may rely on.)

    Also, each command that compiles a file must include the $(COMPILE_PARAMETERS) flag. For example,

    gcc -c $(COMPILE_PARAMETERS) grep.c
    or
    gcc -o grep.exe $(COMPILE_PARAMETERS) grep.c

    This may be used by experiment scripts to tailor the build. For example, it is used by some scripts to instantiate faulty versions of programs (see Section 8 of the C object handbook for additional details on activating fault seeds).

    A second subdirectory that may exist is called:

    versions.seeded

    This directory contains subdirectories

    v1, v2, v3 ... vK

    There may also be a subdirectory v0.

    In versions.seeded, each vK area contains the .c file and .h files needed to build the version, as well as additional files required to build the version with seeded faults activated.

    Note that we could obtain the same behaviors with the .orig and and .seeded variants by turning all faults off, and thus, it might seem that the .orig variant isn't needed. The main reason for including and keeping the .orig variant, however, is that there may be a need, someday, for the original untouched code to be available. For example, someone might wish to create a new set of seeded faults without being influenced by those that had previously been created.

  3. versions

    An empty directory, used during some experiments to hold either a version, or a directory of versions, used in the experiment.

  4. testplans.alt

    Testing information for objects.

    This directory contains subdirectories v0, v1, ... vK, each of which contains testing information for that version. The testplans.alt directory may also contain a testscript subdirectory.

    To avoid keeping multiple copies of identical files, we have organized the vJ subdirectories into two types of files:

    • General Files: These are .tsl, .frame and .universe files defined below.
    • Link Files: These are files that link to a previous version of general files. These allow the inheriting of similar test data from a previous version, thus preventing multiple copies. This will be clarified by the example discussed below.

    General files present in the vj subdirectories are:

    • tsl specification files (.tsl extension)

      These files are named vK_n.tsl or vK.tsl (K=0,1,2...) for different versions and n=1,2,3... for the different tsl specs. For example, if we are in version v0 and there are two tsl specification files then these files will be named v0_1.tsl and v0_2.tsl.

    • sets of test frames generated from .tsl files (.frame extension)

      These files are named vK_n.frame or vK.frame, generated from vK_n.tsl or vK.tsl files respectively. Here K=0,1,2.... for different versions and n=1,2,3...for tsl specs. For example, the frame file generated from vK_1.tsl will be named vK_1.frame.

    • STImpL files (.tsl.universe or .cov.universe extension)

      A STImpL file is a file listing test cases (see also section 6.B of the C Object Handbook). A program called "make-test-script" (mts) transforms STImpL files into various types of scripts that can be used to automatically execute the tests or gather traces for the tests. There are two types of STImpL files:

      • vK_n.tsl.universe or vK.tsl.universe where K=0,1,2,... for different versions and n=1,2,3.. for different tsl specification files. These files contain test cases generated from tsl test frames.
      • vK_n.cov.universe or vK.cov.universe where K=0,1,2.... for different versions and n=1,2,3...for different coverage files. These files contain additional tests to get more code coverage.

    test suite directories (containing multiple test suites, typically these just contain a subset of the lines in some STImpL files). These may be tarred and gzipped due to size. A CONTENT file should identify the meaning of these directories.

    testscript subdirectory
    If there are any startup or exit scripts present in the STImpL file (see 5.B in the object handbook) they are stored in this directory.

    Example

    We use the "flex" object to illustrate the structure of the testplans.alt subdirectory.

    Consider subdirectory v0 in testplans.alt for flex. Here, we find the following files:

    1. General files: v0.tsl, v0.frame, v0.tsl.universe, and v0.cov.universe. These files hold the tsl spec for version v0, the frames generated from that tsl spec, a STImpL file containing tests created from those frames, and a STImpL file containing additional coverage tests, respectively.

    Now if we look at subdirectory v2 under testplans.alt, we find the following files:

    1. Link files. These are v0.tsl, v0.frame, and v0.tsl.universe. These files point to the general files in the v0 subdirectory, as v2 inherits test information from version v0.
    2. General Files. These are v2.tsl, v2.frame, and v2.tsl.universe. These contain the additional tsl specs, frames, and tests for v2, because v2 requires additional testing beyond that performed for v0.

    CONTENTS files should be provided, especially to distinguish the different tsl specs and STImpL files.

  5. testplans

    An empty directory, used during some experiments to hold test suites needed by that experiment.

  6. traces

    An empty directory, used during some experiments to hold test traces.

  7. traces.alt

    This directory contains subdirectories v0, v1, ... vK for the versions of the object; these subdirectories may hold trace information in the form of individual test traces.

  8. inputs

    Files containing inputs or directories of inputs used by programs in various tests. These could be grouped into subdirectories or distinguished by naming conventions if helpful.

  9. outputs

    An empty directory, used during some experiments to hold test outputs.

  10. outputs.alt

    Provides a place to permanently store the outputs of test runs. This is useful for regression testing where new outputs are compared ('diffed') against previous outputs.

    This directory could contain subdirectories v0, v1, ... vK to differentiate between versions. CONTENTS files state what is present.

  11. docs

    Documentation about the object. Subdirectories may be present when useful, such as v0, v1, ... vK directories.

  12. originals

    Original materials obtained on the objects. Subdirectories are used if necessary (e.g. v0, v1, ... vK), and a CONTENTS file will describe the purpose of the subdirectories.

  13. scripts

    The "staging platform" directory from which experiments are run. This directory may also contain saved scripts that perform object-related tasks. Subdirectories can be used if helpful (e.g. v0, v1, ... vK containing scripts for each version).

  14. info

    Other information about the program, especially information gathered by analysis tools and requiring saving for experiments. In particular, we have found it useful to include subdirectories v1, ... vK and "common" to store version-specific and object-specific information, respectively.

    The principle information to be found in the version-specific directories is the fault matrix data, as follows:

    Each vK subdirectory in info may contain fault matrix information for that version. The fault matrices have the following naming convention:

    1. If a fault matrix file is generated from a single STImpL file then it is named fault-matrix.<universe-file>. For example the fault matrix file generated by running the version on the v0_1.tsl.universe file will be named fault-matrix.v0_1.tsl.universe.
    2. If a fault matrix file is generated from the concatenation of two or more STImpL files then it is named fault-matrix.<universe-file-1>.<universe-file-2>>.---.<universe-file-n> where <universe-file-1> is the name of the 1st STImpL file <universe-file-2> is the name of the 2nd STImpL file and so on.

    Fault matrices with .trimmed suffixes are matrices pruned down to include only seeded faults detected by one or more tests, and fewer than 20% of the tests, in the program's STImpL specifications.

    Further information on fault matrices and file names is provided under Section 8 in the C handbook.