SPL Object Handbook

Table of Contents

  1. Overview
  2. Object Selection
  3. Object Organization
  4. General Object Setup
  5. Test Creation
    1. TSL Specification
    2. Test Script Generation
    3. Test Execution
  6. Test Histories and Timings
  7. Fault Seeding and Fault Matrices

1. Overview

This document describes the organization of a typical small to medium size software product line (SPL) as an experiment object. Be aware that some aspects of the preparation process vary depending on the types of analyses and experiments for which the object is expected to be of primary interest. Also bear in mind that an SPL may have associated tools, platform emulators, or pre-processing packages that are required to instantiate the SPL for compilation, product composition, and testing.

The objects in the repository have been designed with support of a certain experimentation workflow in mind. However, we have attempted to create a design that is generic enough to allow customization to other workflows as appropriate. One approach to conducting an experiment with the infrastructure is the following:

  1. Download or create a new copy of one or more objects in a local space for the experiment.
  2. Set the environment variable "experiment_root" to reflect the location of the experiment object. This variable should provide the absolute path to the directory into which you have placed the SIR object directory. This enables scripts and tools provided by the infrastructure to resolve needed paths without hardcoding system specific paths.
  3. Determine the artifacts that you want to utilize, such as specifications, versions, seeded faults, and inputs.
  4. Download any necessary composition, compiliation, or framework packages required by the object.
  5. Create scripts that can leverage the common organizational structure of the objects to perform the desired experimentation tasks on a version of the object, and that can further leverage the infrastructure by relying on common locations for outputs.
  6. Finally, create a top-level script that will perform the experimentation tasks across all versions of an object.

In this way, the object infrastructure facilitates a high degree of automation, enabling experiments that can more rapidly gather much larger amounts of reliable data, in turn leading to results with stronger claims of significance and fewer threats to validity. In some experiments, one can even consider a master script that automatically executes the experiment tasks across multiple objects.

The SIR provides a number of tools to help you conduct experiments on the object programs. The above is only one example of a process for using the infrastructure; many others are possible. However, regardless of the process you choose, you should always perform step 2 above if you intend to use the tools provided by the infrastructure.

2. Object Selection

To populate the repository, we sought SPL objects that satisfied guidelines similar to those applied to our Java objects. We desire that the SPL object possess sufficent complexity so that the products produced by the SPL are themselves non-trival programs. Another aspect we considered in selecting objects is the extent of variablility manifest in the feature model of the SPL. Overly simplistic SPL's were deemed uninteresting because of their limited product compositions (variability in space.) We also sought SPL objects that had multiple versions with new features or capabilities added to each version thus providing an analog to how real−world SPLs evolve to target particular market niches or technological improvements, thus expressing the variability in time property of an SPL.

We prefer SPL objects that generate outputs that can be easily (and automatically) checked by comparison with saved outputs. Deterministic outputs are of paramount concern and the removal of the causes of non−determinisms from the code has been done to the objects provided. Due to the limited number of SPL objects available for study, some objects may in the future rely upon screen dumps. In these objects the test harness will typically be responsible for the binary file comparison required to verify test results. Programs involving concurrency and thread use are also permissible but care must be exercised in the preparation to enforce deterministic output ordering for comparisons.

3. Object Organization

Objects in the repository are organized into a format that supports experimentation activities, which is described here:

SPL object directory specification

This specification is based upon the directory hierarchy of the Java objects provided by the SIR with additional structure to support specifying the composition of each SPL product.

4. General Object Setup

Each SPL source collection has been packaged in a fashion to allow instantiation of the products that can be composed from the available components. Scripts and README files are provided to perform or describe the tasks required to compose the SPL into the products. Some objects may as well provide object specific configuration files tailored to make each product, usually to enable automated production and testing.

As noted earlier, SPL objects may have dependencies on tools or frameworks that may not be delivered within the SIR object package. README files created by the preparer should indicate the source of these external deliverables and indicate how these are used in the SPL composition and testing procedures. Whenever possible these ancillary packages are packaged with the SPL object but due to licensing constraints are left as elements to be collected by the experimenter. Customizations to these supporting components is permissible to remove non−determinism during experimentation as noted below.

SPL objects in the repository may have been modified to provide deterministic output. These modifications are necessary to allow automated regression testing with automated checking of outputs via output differencing, and are motivated in part by the sheer volume of data that is typically generated. In the case of SPL testing, supporting tools and frameworks can add additional sources of non−deterministic output. In these cases the object preparer may provide a modified version of the tool or framework that is modified to remove the non−determinisms or segments of code to be applied to the tool or framework and instructions on how to insert this code. A general treatment of techniques used by test engineers in preparing other SIR test objects can be found in:

Removing sources of nondeterminism

5. Test Creation

We want the test suites for SPL objects in the repository to be constructed using either a TSL specifications methodology or through concrete composition of the abstract cases outlined in use cases provided with the object. The current objects however do not fully reflect this philosophy but instead provide a set of tests that reflect a simplified functional test suite providing minimal coverage. This deficit stems from the limited requirements supplied by the original authors. Normally, because an SPL must provide a set of requirements as part of the necessary non-code assets, the use cases are either derived by the original SPL authors or were created by the SIR object preparer using the requirements provided. Future SIR objects will be required to provide tests suites that reflect a test methodology commonly used with industrial-grade objects.

In general, the tests applicable to a particular version and product of an SPL will have either a TSL specification in the form of a STImpL (universe) file or via provided script files that execute single test frames or a suite of applicable test frames. Notation of the applicable version and product should be included in the file name however some tests from earlier versions or other product compositions may be applicable in keeping with the test reuse philosophy of SPLs.

  1. TSL Specification

    While several legacy objects in the repository support TSL test specifications it should be noted that this type of test specification analysis has not been applied to any of the SPLs currently available from the SIR. The TSL specification was developed to support functional level testing of programs or systems and is applicable to SPLs if the proper level of categorization and partitioning is done to separate the functional units and their constraining parameters or conditions applied to form the test frames. However it should be noted that the STImpL (Simple Test Implementation Language) file format may not support the actions necessary to setup and execute a test of an SPL in which case static test suites are provided instead.

    If a TSL specification is provided with the SPL, the test suites can be created using the mts tool. Execution of the script file generated will run the suite of tests applicable to the version and product that the STImpL file applies to.

    Should you wish to author new or modify existing TSL specifications for objects in the repository, you will first want to familiarize yourself with how TSL specifications are constructed. The process is described in the following paper: "The Category-Partition Method for Specifying and Generating Functional Tests", Thomas Ostrand and Marc Balcer, Communications of the ACM, Volume 31, Number 6, June 1988, pp 676-686. (The paper is available in the ACM Digital Library online). You should then read the documentation describing the specifics of the TSL format that is processed by the tsl tool available from the repository (which is almost the same as the format described in the paper):

    tsl specification

    STImpL file specification

  2. Test Script Generation

    Test scripts for SPL objects that are currently not automatically generated (without STImpL files) provide scripts that execute each test as well as scripts that execute all tests in a suite applicable to the version and product composition. These SPL objects are required to have sufficient information provided through CONTENTS and README files so that a tester can execute these tests as a suite or individually. In general test scripts should be named with information on the version and product to which the script applies, e.g. a file named MM05-13_t1.sh would be test 1 that applies to version 5, product (variant) 13 of the MM product. README or CONTENTS files will provide guidance on the related use case or TSL test frame.

  3. Test Execution

    Test execution on an SPL object is intended to be modular to the single−test level as well as testing in a suite. Tests using JUnit or similar testing frameworks typically are packaged by the original author as a suite for automated regression testing. Our preparation process modifies the way the tests within the test suite are executed so that each test can be executed manually and the outputs captured for pass/fail comparison. When tests are provided via STImpL files, the mts tool creates a shell script containing all tests described in that file. The individual tests can be run manually by extracting only those tests from the mts generated shell script. Since tests from STImpL files can have pre-test and post-test actions, these pre/post sequences need to be extracted along with the test for individual execution.

    For tests that do not use STImpL files a shell wrapping script should be provided that executes all tests that apply to that version and product combination. Extracting individual tests from this arrangement requires the same attention to pre/post activities but typically this has already been done because the shell wrapping script will actually execute other shell scripts that contain the pre/post actions and the test action as a single test frame. Executing the single test frame is accomplished by running this shell script.

6. Test Histories and Timings

Currently none of the SPL objects in the SIR provide test histories. A test history is a mapping from a program code entity (e.g. statement or branch) to the test cases that caused execution of that code. Test histories may be provided in the future for these objects and if so these can be found within the traces.alt directory.

7. Fault Seeding and Fault Matrices

Seeded Faults

SPL objects currently available from the repository are provided with seeded or real faults. As stated in the SPL object directory specification these faulted source versions are found in the directory versions.alt/seeded located under the main directory of the object. The seeded faults are activated either using preprocessor directives that select the faulted code or an overlay method where the original source file is replaced by the faulted version source file. Implementations of seeded faults that use the preprocessor method will have an associated "FaultSeeds.h" file defining the preprocessor symbol that activates that fault when the preprocessor is run on the fault seeded source code. Instructions for building the faulted versions should be part of the README information supplied with the object but a general set of steps required are: copy faulted version into the source directory, run the preprocessor (typically cpp) asserting the fault desired, compile the source into the product variant (this may be via make files, ant build scripts or other methods as defined in the README), and finally execute the tests on the product.

Since all SPL objects currently available from the repository are written in Java, the fault seeding procedures applicable to Java objects can be used to describe the experimental processes and support claims of validity. This process is described in the following document:

SPL fault seeding process

As additional SPL objects are added to the SIR, fault seeding processes for C objects or Java objects may be used.

If your object uses mutation faults, it is expected that the documentation supplied with the object contains sufficient information to allow users to recreate mutated versions of the object. This information will include the name of the mutation tool and links to the publicly available mutation tool. The object should also have scripts that will perform the mutation and/or directions on how to use the mutation tool in order to replicate the mutants. Each mutant version should provide a file that describes the applied mutation, typically in terms that the mutation tool can use in performing the mutation.

Fault Matrices

Fault matrices summarize the relationship of test cases to faults (e.g. they correlate faults to the test cases that reveal those faults). We do not guarantee these fault matrices as authoritative for any platform or environment other than the one under which they were generated. Currently the SPL objects provide fault matrices generated by the original preparer but it is recommended that users generate their own fault matrices following the guidelines expressed in the document:

Java fault matrix generation