GENESIS: Documentation

Related Documentation:

The Developer Package

Maintaining many software projects quickly becomes a lot of work. It also makes it difficult for new people to step into software development, for example, because dependency tracking between software components must be done manually. The Developer package automates the management of multiple software projects. It automates updates to local source code, synchronization between distributed repositories, compilation, testing and installation, packaging and releases, serving monotone repositories and other things. Besides monotone it also has experimental support for integration with mercurial.

The Developer package provides developer utilities that comply with GENESIS development standards. At its core is the neurospaces_build script used for automated software installation and maintenance on a configurable set of software packages. Because this script has many options, the most common operations are provided using simpler frontends. Other scripts are related to version identification of the software, source code documentation comments, testing, and release management. The following sections include simple and detailed examples.

Downloading the Developer Package

The Developer Package can be downloaded from the GENESIS package archive website.

Simple Examples

Simple examples can be used by developers who do not want to know about the intricate details of the Developer tools. Most developers fall into this category. Software developers and laboratories who have added their own private software components should be OK using the simple examples and can skip reading the detailed examples.

Detailed Examples

The detailed examples provide:

• Steps to build a release for the example <your-software> package.
• Examples of the release procedure.

Utilities

Some utilities currently depend on monotone for source code version control due to their configuration. It is possible to work with other version control systems.

Main script

• neurospaces_build: Performs various back-end operations on a set of software packages. It is configurable in its packages and operations. The command:

  neurospaces_build --help

  describes how the script works and lists default actions and build options. The flags at the end of the output should be inspected.

• In brief: neurospaces_build operates in one of two modes determined by where the script finds the source code (“neurospaces_build --help” gives more details). As indicated in the following figure, neurospaces_build creates an extensible associative array (hash table) that can be visualized as a matrix of packages (rows) and build operations (columns). The script traverses the matrix by row, i.e. packages first, and performs successive build operations for a given package. The figure below lists the default build options. Options to neurospaces_build select or deselect cells in the matrix according to the configuration specified by any options to the neurospaces_build command. Note: Addition of a new component to GENESIS is the equivalent of adding a new column to the neurospaces_build associative matrix.

  ---

  ---

• Installation Order of Components: This is the left -> right component order in the neurospaces_build associative array:
  1. Developer Component: Described in this documentation.
  2. Model Container: Provides a highly optimized solver-independent internal storage format for models.
  3. Exchange: Provides exchange in common descriptive standards such as NeuroML and NineML.
  4. Experiment: Experimental Protocols.
  5. Heccer: The current standard numerical solver.
  6. SSP: The simple Scheduler in Perl is currently the standard scheduler for GENESIS.
  7. NS-SLI: The GENESIS 2 Backward Compatibility Module. Required to run GENESIS 2 scripts.
  8. G-Shell: Runs interactive GENESIS sessions.
  9. GENESIS Documentation System: A modular documentation system designed to maintain geographically distributed software systems.
• Note: When installing GENESIS manually, it is recommended to follow the installation order given above.

Front-end Developer Scripts

The following utilities take optional arguments of “--regex” to select the packages they operate on, and “--verbose” to run the command in a more verbose mode. They follow the order of the Developer Workflow. The information they return are determined by the configuration of the Developer package.

Note: These Developer package commands are run from a command line prompt, i.e. outside the GENESIS environment.

1. Write/Update Code:
   • neurospaces_create_directories Create the correct directory layout required for GENESIS development.
   • neurospaces_download Download GENESIS components from a central archive.
   • neurospaces_init Create source code repositories.
   • neurospaces_pull Download the source code or patches from a remote repository. This places the newest version of the source code in your local repository.
   • neurospaces_push Upload source code or patches to a remote repository.
   • neurospaces_serve Starts serving the source code repositories such that other people can pull and sync to your machine (note that this locks all your local databases).
   • neurospaces_status Check for local source code modification (no network required).
   • neurospaces_sync Synchronize local source code modification with a repository.
   • neurospaces_update Makes the local source code up to date using the repositories locally stored on your PC (so this is an entirely local operation). If necessary, this command also automatically merges parallel development branches.
   • neurospaces_upgrade Pulls code from the GENESIS repositories, calls neurospaces_update and neurospaces_install
   • neurospaces_tools_propagate Propagates updates to source code of shared tools from source to target packages.
2. Document Code API:
   • mcad2doxy Convert obsolete multicad documentation to doxygen format, this has been used to convert the developer documentation of some of the software components to doxygen format.
   • neurospaces_docs Builds package specific documentation on your local machine.
   • neurospaces_website_prepare Prepare a version of the website on your local PC, and optionally upload it. This command builds a subset of documentation maintainted with the command neurospaces_docs.
3. Build/Check:
   • neurospaces_configure (Re)configures the simulator software (requires packages to be installed already).
   • neurospaces_install Install new simulator executable binaries and scripts.
   • neurospaces_uninstall Uninstalls the simulator software (including the Developer package commands, to reinstall, go to the Developer package source code directory and run “make && sudo make install”).
   • neurospaces_clean Clean local source code and documentation directories.

   The neurospaces_upgrade command makes the local source code up to date with the source code served by the remote servers. It thereby combines steps 1 and 3 explained above. This command is similar in operation to the invocation of the commands neurospaces_uninstall, neurospaces_pull, neurospaces_clean, neurospaces_update, neurospaces_install

4. Regression Testing:
   • nstest_query Queries test specifications of a package.
   • neurospaces_harness Run the test specifications found in the current directory, see also the Neurospaces tester.
   • neurospaces_check Check for correctness of the installed software.
   • Simple Example: Enable the distcheck target of the make files, in addition to the default actions:

     $ neurospaces_build --distcheck --help

5. Internal/Public Release:
   • nspkg-deb Automated debian package builder.
   • neurospaces_packages Show which software components are enabled on your local machine.
   • Simple Example: This command invokes

     $ neurospaces_build --help-packages

   • neurospaces_dist Build a release given a milestone label and version number. This script also inserts the milestone label and version number into the Tag database which is a YAML file and can be found at: /etc/neurospaces/tag_database. The following two utilities query this database:
     • td-labels Shows all release labels (major and version numbers).
     • td-majors Shows all major release labels.

     For updating of version keywords the following are invoked automatically by the neurospaces_build script:

     • release_extract Extract release information from a monotone repository. When a Tag is set, that will be the result, otherwise the SHA of the current base revision will be the result.
     • release_expand Do keyword expansion, see the manpage in the source for more details.
   • Simple Example: What shell commands will be run after options have been given, without really running the commands.

     $ neurospaces_build --verbose --distcheck --dry-run --developer

   • Simple Example: How are operations mapped to packages.

     $ neurospaces_build --help-all

   • Detailed Example: Steps in building a release for the example <your-software> package.
     • Tag the code: The GENESIS convention for creating these tags, which must be unique, is to concatenate an identifier from the name of the lab of origin, the purpose and/or type of software (e.g. build, passive, active, python, userdocs, des, network, i64, purkinje, integration, pools, etc), with an appended numerical identifier. Here, for example we use the generic tag “<mylab-mysoftware-vnum>”.

       $ neurospaces_build --tag <mylab-mysoftware-vnum> --verbose --verbose --verbose --no-compile --no-configure --no-install --regex <my-software> --developer

     • Build the release for the tagged code:

       $ neurospaces_build --dist --verbose --verbose --verbose --no-compile --no-configure --no-install --regex <my-software> --developer

     • Or with one command line:

       $ neurospaces_build --tag <mylab-mysoftware-vnum> --dist --verbose --verbose --verbose --no-compile --no-configure --no-install --regex <my-software> --developer

     • Upload the tarballs:

       $ neurospaces_build --src-tag <mylab-mysoftware-vnum> --upload-server ftp://upload.sourceforge.net/incoming --verbose --verbose --verbose --no-compile --no-configure --no-install --regex <my-software> --developer

       Do not forget to edit release notes, tag the files as “Any”, “Source.gz”, etc. Due to the crappy Sourceforge interfaces, this has to be done manually (anyone have any ideas?).

6. Integrate Software Components:
   • neurospaces_versions Shows which versions of the GENESIS packages are installed.
   • neurospaces_cron A cron job based tester script.
   • Simple Example: What operations on known packages exist.

     $ neurospaces_build --help-operations

Detailed Examples of the Release Procedure

After putting a Tag on the code, you first must check if the package can be built from the tarball. The autotools distcheck target does this (see below). An additional check is to build the tarball, put it somewhere on your filesystem, and then build the check target from the tarball. For example, first build the tarball and put it somewhere on your filesystem (e.g. /tmp/uploads):

Then build the check target from the tarball:

In a one line summary you need to insert a --tag, and --distcheck the current code, then make them available on Sourceforge. In one line of code:

If you don’t want to do the upload, for a single package:

If that works, you then want to do the upload:

The --verbose option is there to let you know what is going on during this lengthy procedure.

The same procedure is used to build a release of either individual or all packages. An official public release is made available for download on Sourceforge. Intermediate releases are for internal use only.

Because official releases are built using automake’s distcheck target, they must pass the tests of the package (on the machine used for the build). So they are considered to be alpha releases (ie. internally well tested).

The release procedure normally checks for package correctness using the automake distcheck target, meaning that both install and uninstall targets work and are each other’s complement. Official releases must always succeed on this target. Intermediate releases can fail.

Other Examples

• Uninstall 4 packages on your developer machine:

  $ neurospaces_build --verbose --verbose --verbose --no-compile --no-configure --uninstall --regex ’(my-software|experiment|heccer|ssp|studio)’ --developer

• Build checked releases for four packages:

  $ neurospaces_build --verbose --verbose --verbose --no-compile --no-configure --no-install --distcheck --regex ’(my-software|experiment|heccer|ssp|studio)’ --developer

• After a modification of the developer package, reinstall it:

  $ neurospaces_build --enable developer --regex developer --developer --verbose --verbose --verbose

• Releasing the developer package on Sourceforge:

  $ neurospaces_build --tag build-25 --dist --src-tag build-25 --upload-server ftp://upload.sourceforge.net/incoming --enable developer --regex ’(developer)’ --developer --verbose