GENESIS: Documentation

Neurospaces Tester

The Neurospaces tester framework provides a general purpose framework for regressing testing.

It includes:

Testing the output and behavior of both simple and complex applications.
A human readable report of the tests performed, a description of the errors (if any), and configuration options and environment.
Testing application command line options.
Testing communication over file descriptors with an application, and testing of the behaviour of an application.
Dynamic reconfiguration and recompilation of the tested software.
Testing the consistency of the source code.
Built in support for application specific libraries.
Interaction with the environment, including environment variables.
Running the application in a chrooted environment.
Conversion of the test cases to html.
A human readable description of the functions provided by the tested application.

Despite the wealth of functionality, using and configuring the tester is quite simple (see below for more). The latest version of the tester framework can be found in the DeveloperPackage. The same framework is used for regression testing of all GENESIS related software.

The Neurospaces tester is written in Perl. Knowledge of Perl is not necessary to be able to work with the tester. Basic knowledge of Perl might be handy, and is now explained.

Tester Calling

The most convenient way to call the tester is by doing “make check”. For GENESIS software packages, this calls neurospaces_harness, a Perl script that runs all the tester specifications found in tests/specifications.

neurospaces_harness: Runs all test specifications found in the specifications directory. A test specification ends in the suffix .t.
--regex: This option runs those test specifications that match with the given regular expression. For example
neurospaces_harness --regex convertors

runs all tests whose pathnames match with convertors.
--numerical-compare: Enables numerical comparisons for numerical output.
--randomize-order: Randomizes the order of the tests before executing them.
--timeout-multiplier: Multiplies all timeout values with the given constant.
--timings: Adds timing information to the test report.

Tester Configuration

The Neurospaces tester needs to know certain things about your machine before tests can be executed. These things are in the tester configuration file, named tests.config, located in your current directory.

It is normal that a software package has multiple tests.config files, e.g. in several directories.

The tests.config file defines a Perl hash (we plan to also support JSON). In the test specifications, this hash is available in the variable main::config. So to access the core_directory entry in the hash, you have

$main::config->{core_directory}

Here is an example script:

#!/usr/bin/perl -w

use strict;

my $package_name = "heccer";
my $package_label = "build-8";
my $package_version = "build-8-0";

my $monotone_id = ‘mtn automate get_current_revision_id‘;
chomp $monotone_id;

my $config
    = {
         core_directory => ’./’,
         description => ’Configure the tester when run from this directory’,
         outputs_dir => ’./tests/html’,
         package => {
            label => $package_label,
            name => $package_name,
            version => $package_version,
            version_control_id => $monotone_id,
         },
         tests_directory => ’./tests/specifications’,
      };

return $config;

Definitions

core_directory: The directory that contains the source code of your package. Tests can use this directory to read files.
description: Short (few words) description of the tests.
outputs_dir: Defines the directory where the htmlified test specifications (generated using tests_2_html) are located.
package: Contains package version information.
tests_dir: Contains the directory to be searched for test specifications with the .t suffix.

Tester Specifications

A tester specification is contained in a file with a name that has a suffix of .t. The tester specification tells the tester what application to run, with what command line, command line options, and under what environment. It is possible to change directory before the application gets executed, or to change to a chrooted environment.

After running the application, the tester communicates with the application over stdin/stdout/stderr (general socket communication under development). The output read from stdout/sterr is checked with expected output (literal output or regular expressions), and mismatches are reported.

Multiple writes to and reads from the application are possible.

If, for whatever reason, the same application is started in more than one test specification, the running application is ‘recycled’ (unless the side_effects option is used, see below).

If there was configuration in effect for an application specific library, the library is checked for changes after a test has been performed. Changes to the library are not allowed.

Test Specifications

A test specification is a file with Perl code that returns a Perl hash (as noted above, we plan to also support JSON):

#!/usr/bin/perl -w
use strict;
my $test
= {
. . .
. . .
};

return $test;

The content of the hash ref encodes all the tests in the specification. The following example is taken from the Model Container software package:

#!/usr/bin/perl -w
#
use strict;
my $test
    = {
         command_definitions => [
         {
            arguments => [
               -q’,
            ],
            command => ’./neurospacesparse’,
            command_tests => [
               {
                  description => "Is neurospaces startup successful ?",
                  read => ’neurospaces ’,
                  timeout => 3,
                  write => undef,
               },
               {
                  description => "Does the version information match with \
                     model-container-build-8 ?",
                  read => "model-container-build-8",
                  write => "version",
               },
            ],
               description => "check version information",
               },
            ],
            description => "run-time versioning",
            name => ’version.t’,
         };
return $test;

A test specification contains the following keys/values:

comment A general purpose comment, e.g. this test is broken because ….
description Describes the purpose of the test in a few words (not an entire paragraph).
name Must be the same as the filename, e.g. version.t.
command_definitions An array ref with hash refs. Each hash ref contains the tests for one application run. See below.

Command Definitions

Arguments and command keys are the shell command line to run, and to connect stdin and stdout.
command_tests See below.
description Gives the purpose of the test in a few words (not a paragraph).

Command Tests

The description key is mandatory, the other keys are optional.

description Describes the purpose of the test in a few words (not an entire paragraph).
read Expected output from the application.
- If it is a string, it is taken as (a part of the) output to be expected literally.
- If it is an array, the first element in the array should be -re, while the second element can be a regular expression that is matched against the real output of the application.
- If it is a hash, the hash must contain alternative keys that enumerate alternatives for the expected output in an array.
write Writes output to stdin of the application before any output is generated.
comment A text comment.
side_effects Avoids recycling of application commands, even if the options are the same.
preparation and reparation Two commands to bring the application environment into a particalur state and restore the previous state after the test has been performed. Do not use this to change the application state, use the write key for that.
disabled Explains why a test has been disabled. Omitting this key or setting this key to a false value or the emtpy string, enables the test.

Tester HTML

UNDER CONSTRUCTION

Tester Tutorial

UNDER CONSTRUCTION