PGENESIS 2.2 Installation and Use Nigel Goddard & Greg Hood Pittsburgh Supercomputing Center pgenesis@psc.edu (Updated January 15, 2002 for PGENESIS 2.2a) PGENESIS 2.2a incorporates some bug fixes to the July 2001 2.2 version. NOTE: Before installing this software, please check that you have the latest version by checking the GENESIS ftp site: ftp://genesis-sim.org/pub/genesis/. We *strongly* advise you to download the latest version if you don't already have it. Additional information about PGENESIS may be obtained from the PGENESIS home page: http://www.psc.edu/Packages/PGENESIS The GENESIS email address is now genesis@genesis-sim.org, the GENESIS web page is now located at http://www.genesis-sim.org/GENESIS, and the ftp site is now ftp://genesis-sim.org/pub/genesis/. INTRODUCTION ------------- PGENESIS (Parallel GENESIS) can be run on a single workstation; the PVM software system is used to provide the illusion of a parallel computer. It is much easier to run on a network of workstations if they have a shared file system so that any workstation can access any file. If this is not the case, you will need to install PGENESIS independently on each workstation and you will need a copy of the files needed for your simulation (data files, script files, etc), preferably in an identically named directory path. PGENESIS adds a library (parlib) to an installed version of GENESIS, creating an executable "pgenesis" that has the usual GENESIS objects and commands, plus a "postmaster" object (post) and some other new objects and commands. PGENESIS 2.2 is an update of version 2.1 that has the changes needed to work with GENESIS version 2.2, as well as a few bug fixes and changes needed to compile with recent OS versions. PGENESIS 2.1 will not work with GENESIS 2.2. This document describes how to install PGENESIS by compiling from source code. INSTALLING PGENESIS ------------------- 1. Compile and install serial GENESIS version 2.2 If you have not obtained GENESIS already, download it from the GENESIS ftp site: ftp://genesis.bbb.caltech.edu/pub/genesis/genesis2.2.tar.gz or copy from the CDROM if you have a CDROM distribution. Henceforth we will refer to the top level directory of GENESIS as ~genesis, and the top level directory of PGENESIS source as ~pgenesis. This is a gzipped tar file which can be extracted with: cat genesis2.2.tar.gz | gunzip | tar -xvf - Follow the instructions in ~genesis/src/README. Make sure that after doing a "make" or "make nxall" you also do a "make install" or "make nxinstall". 2. Ensure PVM is installed If you are not sure about the status of PVM on your system, check with your system administrator. You should have version 3.3.4 or later. If pvm is already on your PATH you can find out the version number thus: echo version | pvm If you need to install PVM, you can get the latest version, as well as more information about PVM from the PVM home page: http://www.epm.ornl.gov/pvm/pvm_home.html. You do not need the root password to install PVM. After untarring the distribution, you will find installation instructions in pvm3/Readme. A PVM manual is is also available from the PVM home page. A PVM distribution and manual is also available from the GENESIS ftp site, but it is not guaranteed to be the most recent version. Additionally, you may need to compile PVM even if you already have an executable file, since the object libraries may be missing in some PVM installations. 3. Compile the parallel libraries and executables The Makefile.dist file in the ~pgenesis directory should be copied to Makefile, which should then be edited for your platform type as described by the comments in the Makefile. Note that this is NOT ~pgenesis/src/Makefile, which should be left as is. You must set PVM_ROOT and PVM_ARCH in the Makefile, unless you have set them in your .cshrc as described in the PVM manual. The default setting for PVM_ARCH should be correct in almost all cases. You will have to uncomment one of the machine/compiler dependent options for CC, CPP, CFLAGS, LIBS, SYSLIBS, etc. The GENESIS and INST_DIR macros should be set to, respectively, the full pathnames of the serial GENESIS installation and the directory in which parallel GENESIS should be installed (usually the PGENESIS root directory). The PARSRC_DIR should be set to the full pathname of ~pgenesis. Finally, uncomment the EDITED=yes assignment; the Makefile will not function until this is done. Makefile lists several different options to be used with "make": 'make nxinstall' - to compile and install pgenesis without Xodus 'make install' - to compile and install pgenesis with Xodus 'make extended' - to install extended pgenesis with Xodus 'make nxextended' - to install extended pgenesis without Xodus 'make env' - to show some of the configuration settings 'make clean' - to remove this machine type's object files 'make spotless' - to remove all machine types' object files Normally, you will use the 'make install' command, which compiles and installs PGENESIS with Xodus. Once the Makefile has been customized, execute: make install If you do not use Xodus, first make nxgenesis for serial GENESIS, and then execute "make nxinstall". If there are errors that are not ignored, then redo this command piping the output into a file and submit it as a bug report to genesis@bbb.caltech.edu: make install > & ! make.out & (or for the Bourne shell: make install > make.out 2>&1 & ) Note that, unlike the case with serial GENESIS, "make install" or "make nxinstall" is used to both compile and install PGENESIS. This will create a shell script bin/pgenesis, which can be executed in order to start PVM and run the proper pgenesis binary file. The pgenesis binary is created in bin/, where is one of the architectures supported by PVM (e.g. ALPHA, LINUX, SGI5, SUN4, T3E, etc.) It will also create a .psimrc file that should be copied to your home directory. RUNNING PGENESIS ----------------- Each user should be make sure that the PATH is set to include ~pgenesis/bin, where ~pgenesis is the root of the PGENESIS installation, i.e., the INST_DIR variable in the top level Makefile. If the script files you wish to use are not in your home directory, the examples directory, or the directory in which you run parallel GENESIS, you will need to add their directory path to the SIMPATH variable in the .simrc file you use. By default this is .psimrc for PGENESIS worker nodes. You can run PGENESIS by typing "pgenesis scriptfile". There are several available options, which may be listed by typing "pgenesis -help". The most commonly used of these are: -config where is a file containing a list of machines to be used (1 per line) -nox run without Xodus -debug tty create a window for each pgenesis node -browse browse the documentation using netscape FURTHER DOCUMENTATION --------------------- For any use of PGENESIS beyond the simple test described below, see the documentation in the Doc and Hyperdoc directories. You can access the HTML documentation by pointing your browser at Hyperdoc/welcome.html. The README.mp file gives instructions for use on multiprocessor machines, and README.paragon is specific to the Intel Paragon (as installed at Caltech). The documentation also describes the use of PGENESIS on a network of workstations. TESTING PGENESIS ---------------- After copying .psimrc to your home directory, set your path to include ~pgenesis/bin, in order to be able to execute the pgenesis script from any directory. For example, if the Makefile setting for PARSRC_DIR is /usr/local/pgenesis and your shell is tcsh or csh, use: set path = ($path /usr/local/pgenesis/bin) If you are using bash as your shell, use PATH=$PATH:/usr/local/pgenesis/bin You can then do a simple test of PGENESIS by running the demo.g simulation script in the pgenesis Scripts/orient1 directory. After changing to this directory, start it with "pgenesis demo.g". This is a 3-way parallel decomposition of the genesis orient_tut simulation over 3 nodes. The default is to create these nodes on a single processor. (Of course, to achieve any advantage of using PGENESIS, you will want to use networked workstations or a multiple processor machine. Details for doing this are given in the documentation.) The display will be somewhat like the serial GENESIS orient_tut simulation, and clicking on "sweep_horiz" should produce a similar pattern of activity and plots. A log file is created in o.out, so you will need to have write permission in the directory that contains this simulation. For an even simpler test, or for debugging, you can try Scripts/hello.g. DEBUGGING PGENESIS SCRIPTS --------------------------- Normally the master node sends its output to your screen and the worker nodes send their output to the file /tmp/pvml.???? on the machine they are running on, where ???? is your uid on that machine. Many script bugs can be tracked down by examining those /tmp/pvml.???? files. Note that PVM appends to these files on every run, until PVM is halted and restarted. Also note that if a worker node exits without executing "paroff", some of its output to stdout/stderr which is buffered may not appear in the /tmp/pvml.???? file. Output from the workers can be redirected to a file by giving the paron command the -output option. can be /dev/tty, in which case the worker output appears on your screen mixed with the master's output. If you are running X-Windows, you can have each worker create its own window and send its output there by using the "-debug tty" flag when invoking PGENESIS. This requires that your DISPLAY environment variable is set to "hostname:0" when you invoke the pgenesis script. By default PGENESIS starts the workers using -silent 3 which minimizes the messages printed by the workers (e.g., the banner). To see more worker status messages, give the paron command the -silent 0 option. Common errors include: 0. User impatience leads to interruption of PGENESIS before it has completed the initialization of the workers. The timeout is set to 120 seconds, for maximum information about the problem, let PGENESIS timeout if it fails to initialize workers properly (i.e., wait!). 1. PVM can't find the worker executable. This is specified with the -executable option of the paron command, the default is "pnxgenesis". This executable, or a link to it, should be in ~/pvm3/bin/$PVM_ARCH. A symptom of this problem is that PGENESIS reports it failed to spawn workers and the /tmp/pvml.??? files are empty apart from the header. 2. PVM aborts worker startup due to I/O in your .cshrc Symptoms of this are that PGENESIS reports workers failed to spawn and the /tmp/pvml.??? files contain error messages about TTY or other strange things, and no GENESIS startup banner. Modify your .cshrc as described in pvm-cshrc.stub. 3. The workers can't find the script containing the paron command. Symptoms of this are that PGENESIS reports workers failed to spawn and the /tmp/pvml.??? files show the GENESIS prompt with no attempt to execute paron, or the startup banners show that the script file could not be found. Edit your ~/.psimrc and ~/.pnxsimrc so that SIMPATH lists the directories containing the scripts. 4. Incorrect use of barriers. Symptoms are that the master or some worker(s) hang and eventually timeout. Examine your use of barriers. Note that some commands contain implicit barriers, it is possible there is a bug in the source - if you suspect this mail ngoddard@psc.edu. For more information on the finer points of using PVM, refer to the PVM user guide. USER-EXTENDED PGENESIS ----------------------- Libraries are constructed just as for serial GENESIS, each in its own subdirectory in a canonical form. See the GENESIS documentation for descriptions of this form. In this section we describe how to link these libraries with PGENESIS components to create an extended PGENESIS. Assuming your libraries are subdirectories of a root directory I'll call MYPGEN, these are the steps you follow to create and run extended PGENESIS. A. Copy ~pgenesis/lib/Usermake to MYPGEN/Makefile. Modify the Makefile for your platform type as described in the Makefile. You must set PVM_ROOT and PVM_ARCH in the Makefile, unless you have set them in your .cshrc as described in the PVM manual. The default setting for PVM_ARCH should be correct in almost all cases. You will have to uncomment one of the machine/compiler dependent options for CC, CPP, CFLAGS, LIBS, SYSLIBS, etc. The GENESIS and INST_DIR macros should be set to, respectively, the full pathnames of the serial GENESIS installation and the directory in which parallel GENESIS should be installed (usually the PGENESIS root directory). B. Now add the names of your libraries and their subdirectories to USERLIB and USERDIR and set the installation root for your extended PGENESIS in USERINST. Execute 'make nxextended' to make extended PGENESIS without Xodus 'make extended' to make extended PGENESIS with Xodus. C. This should create USERINST/bin and USERINST/lib containing various files. The only one you need to be aware of is USERINST/bin/pgenesis. To run your extended PGENESIS, set your path to include USERINST/bin early. Then invoke extended PGENESIS with 'pgenesis' just as you would for vanilla PGENESIS.