SPL Object Preparer's Handbook
Table of Contents
- Overview
- Object Selection
- Object Organization
- General Object Setup
- Test Creation
- Test Histories and Timings
- Fault Seeding and Fault Matrices
1. Overview
This document describes the procedures followed in the preparation of software product line (SPL) as an experimental research object. These instructions are intended for those who set up an SPL or extend an existing SPL artifact but also serve to describe the SIR packaging structure and discuss the choices available during the preparation process. This preparation process has been puposely generalized due to our limited exposure to and the limited availability of SPL test subjects so far. As with the C and Java test objects in the SIR, some aspects of preparation may vary depending upon the types of analyses and experiments that are anticipated to be of interest when testing that SPL.
2. Object Selection
Our requirements for SPLs are less stringent than for previous artifacts in the repository due to the aforementioned limited availability of open source SPL projects. Any candidate SPL needs to provide at minimum a set of core and optional features that can be composed into distinct products (variants) and not be overly simplistic. Tractable size should also be considered since too many products (or too large of source code base) can cause testing and analysis efforts to become beyond a reasonable level for empirical studies.
A good guideline to judge an SPL for suitability is the number of products it can ultimately compose. This number of products may be much larger than the envisioned set of products that the original authors defined in their product development process because all possible combinations of features may not have been deemed viable or interesting products. Existing SPL subjects that we have packaged are relatively small (1.5 KLOC) but provide sufficient variability to compose more than 30 products. Additionally, an SPL with a wealth of potential product combinations is a desirable test subject but may require the end-user to limit the scope of products studied to accommodate time or resources limits.
3. Object Organization
Objects in the repository are organized into a format that supports experimentation activities, which is described here:
SPL object directory specification
This specification is based upon the directory hierarchy of the Java objects provided by the SIR.
4. General Object Setup
Each SPL source collection has been packaged in a fashion to allow production of the products that can be composed from the available components. Scripts and README files are provided to perform or describe the tasks required to compose the SPL into the products. Some objects may as well provide object specific configuration files tailored to make each variant, usually to enable automated production and testing.
As noted earlier, SPL objects may have dependencies on tools or frameworks that may not be delivered within the SIR object package. README files created by the preparer should indicate the source of these external deliverables and indicate how these are used in the SPL composition and testing procedures. Whenever possible these ancillary packages are packaged with the SPL object but, due to licensing constraints, may be left as components to be collected by the end user (experimenter). Customizations to these supporting components is permissible to remove non−determinism during experimentation as noted below.
SPL objects in the repository may have been modified to provide deterministic output. These modifications are necessary to allow automated regression testing with automated checking of outputs via output differencing, and are motivated in part by the sheer volume of data that is typically generated. In the case of SPL testing, supporting tools and frameworks can add additional sources of non−deterministic output. In these cases the object preparer may provide a modified version of the tool or framework that is modified to remove the non−determinisms or segments of code to be applied to the tool or framework and instructions on how to insert this code. A general treatment of techniques used by test engineers in preparing other SIR test objects can be found in:
Removing sources of nondeterminism
5. Test Creation
We want the test suites for SPL objects in the repository to be constructed using either a TSL specifications methodology or through concrete composition of the abstract cases outlined in use cases provided with the object. The current objects however do not fully reflect this philosophy but instead provide a set of tests that reflect a simplified functional test suite providing minimal coverage. This deficit stems from the limited requirements supplied by the original authors. Normally, because an SPL must provide a set of requirements as part of the necessary non-code assets, the use cases are either derived by the original SPL authors or were created by the SIR object preparer using the requirements provided. Future SIR objects will be required to provide tests suites that reflect a test methodology commonly used with industrial-grade objects.
In general, the tests applicable to a particular version and product of an SPL will have either a TSL specification in the form of a STImpL (universe) file or via provided script files that execute single test frames or a suite of applicable test frames. Notation of the applicable version and product should be included in the file name however some tests from earlier versions or other product compositions may be applicable in keeping with the test reuse philosophy of SPLs.
- TSL Specification
While several legacy objects in the repository support TSL test specifications it should be noted that this type of test specification analysis has not been applied to any of the SPLs currently available from the SIR. The TSL specification was developed to support functional level testing of programs or systems and is applicable to SPLs if the proper level of categorization and partitioning is done to separate the functional units and their constraining parameters or conditions applied to form the test frames. However it should be noted that the STImpL (Simple Test Implementation Language) file format may not support the actions necessary to setup and execute a test of an SPL in which case static test scripts (and suites) are provided instead.
If a TSL specification is provided with the SPL, the test suites can be created using the
mtstool. Execution of the script file generated will run the suite of tests applicable to the version and product that the STImpL file applies to.Should you wish to author new or modify existing TSL specifications for objects in the repository, you will first want to familiarize yourself with how TSL specifications are constructed. The process is described in the following paper: "The Category-Partition Method for Specifying and Generating Functional Tests", Thomas Ostrand and Marc Balcer, Communications of the ACM, Volume 31, Number 6, June 1988, pp 676-686. (The paper is available in the ACM Digital Library online). You should then read the documentation describing the specifics of the TSL format that is processed by the tsl tool available from the repository (which is almost the same as the format described in the paper):
- Test Script Generation
Test scripts for SPL objects that are currently not automatically generated (as is the case with other SIR objects that provide TSL specifications in the form of STImpL files) do provide scripts that execute each test as well as scripts that execute all tests in a suite applicable to the version and product composition. Thus SPL objects contained in the repository are required to have sufficient information provided through CONTENTS and README files so that a tester can execute these tests as a suite or individually. In general individual test scripts should be named with information on the version and product to which the script applies, e.g. a file named MM05-13_t1.sh would be test 1 that applies to version 5, product (variant) 13 of the MM product. README or CONTENTS files will provide guidance on the related use case or TSL test frame. Scripts that execute all tests applicable to a version and product should be named in a similar fashion containing a descriptive name, e.g. runall_v5-13 runs all tests on version 5 product 13. Since test reuse is a benefit of an SPL, the idividual tests run by the "runall" script may reference earlier versions and/or products.
7. Test Histories and Timings
Currently none of the SPL objects in the SIR provide test histories.
A test history is a mapping from a program code entity (e.g. statement or branch)
to the test cases that caused execution of that code. Test histories may
be provided in the future for these objects and if so these can be found within
the traces.alt directory.
8. Fault Seeding and Fault Matrices
Seeded Faults
SPL objects currently available from the repository are provided with
seeded or real faults. As stated in the SPL object directory
specification these faulted source versions are found in the directory
versions.alt/seeded located under the main directory of the
object. The seeded faults are activated either using preprocessor directives
that select the faulted code or an overlay method where the original source
file is replaced by the faulted version source file. Implementations of seeded
faults that use the preprocessor method will have an associated "FaultSeeds.h"
file defining the preprocessor symbol that activates that fault when the preprocessor
is run on the fault seeded source code. Instructions for building the faulted
versions should be part of the README information supplied with the object but
a general set of steps required are: copy faulted version into the source directory,
run the preprocessor (typically cpp) asserting the fault desired, compile the
source into the product variant (this may be via make files, ant build scripts
or other methods as defined in the README), and finally execute the tests on
the product.
Since all SPL objects currently available from the repository are written in Java, the fault seeding procedures applicable to Java objects can be used to describe the experimental processes and support claims of validity. This process is described in the following document:
As additional SPL objects are added to the SIR, fault seeding processes for C objects or Java objects may be used.
Fault Matrices
Fault matrices summarize the relationship of test cases to faults (e.g. they correlate faults to the test cases that reveal those faults). We do not guarantee these fault matrices as authoritative for any platform or environment other than the one under which they were generated. Currently the SPL objects provide fault matrices generated by the original preparer but it is recommended that users generate their own fault matrices following the guidelines for Java fault matrices, expressed in the document: