PGENESIS 2.0 Manual

    ###
    ### WARNING: This document describes the obsolete PGENESIS 2.0, which
    ###          dates from around 1995-1996; users should refer to the
    ###          top-level README, the PGENESIS homepage
    ###          (http://www.psc.edu/Packages/PGENESIS), and the online
    ###          Book of GENESIS (http://www.genesis-sim.org/GENESIS/bog/bog.html
    ###          for current information.
    ###
    ###          This is included here for its historical relevance as a
    ###          design document, and as a supplementary source of information
    ###          for users seeking additional information about various PGENESIS
    ###          features.
    ###

    	Script Language Extensions for Parallel Genesis

    			Reference Manual
    			 Nigel Goddard
    			ngoddard@psc.edu

    ===================================================================
    1. Introduction

    Parallel Genesis allows a modeler to distribute a simulation across a
    compuational platform that supports the PVM message passing library.
    This includes most supercomputers and workstation networks.  There is
    a natural mapping from network-level models to a parallel computer,
    but Parallel Genesis is also capable of executing a single-cell model
    on a parallel platform.

    This document describes the language extensions for parallel programming
    in the Genesis script language.  The first section introduces the
    programming model and the second section specifies the syntax and
    semantics of each new script language extension.

    1.1 PVM and Parallel Genesis

    The Parallel Virtual Machine (PVM) system provides the illusion of a
    parallel platform.  The PVM system may run on a single CPU or multiple
    CPUs, possibly of different type.  In the rest of this document when
    we refer to the "parallel platform" we mean the illusion provided by
    PVM.  When we refer to the "parallel machine", we mean the physical
    set of CPUs and network connecting them that PVM is running on.  An
    executing PVM program consists of user processes, typically one per
    CPU, which communicate via the PVM daemon which runs on each
    participating CPU.  In parallel Genesis, each user process is an
    independent Genesis simulation, which we call a "node" of the parallel
    simulation.  Nodes are uniquely identified by a node number
    (consecutive integers starting at zero).

    1.2 Zones

    Nodes can be grouped in "zones" when the simulation is started.  Each
    node is in exactly one zone (by default, every node is in zone 0).
    The zones form a fixed partition of the parallel platform.  The
    motivation for zones is to allow different parts of the simulation to
    run asynchronously (uncoordinated).  For example, in a parameter
    search application one might wish to run many instances of a four-node
    model in parallel.  Each instance uses four nodes which must run
    synchronously, but the instances need not be coordinated (except at
    start and finish).  Thus we can run each instance in a separate zone,
    each zone containing four nodes.  Zones are uniquely identified by
    consecutive integers starting at zero.  The nodes within a zone are
    uniquely identified with a znode number (consecutive integers starting
    at zero).  The mapping from node numbers to (zone, znode) pairs is
    discussed in section ???.

    1.3 Programming model

    1.3.1 Namespace (memory)

    The Parallel library currently provides a private-namespace
    programming model.  This means that each node has no knowledge of the
    elements that reside on other nodes.  This implies that every
    reference to an element on another node must specify the node
    explicitly.  It is envisioned that a shared-namespace programming
    model will be implemented eventually.  This will allow nodes within a
    zone to reference elements on other nodes in the zone without
    specifying the node number.  To ease upgrade of parallel models to the
    shared-namespace paradigm, it is recommended that element names within
    a zone be unique.  If this recommendation is not adhered to, there
    will be naming conflicts if a model wishes to take advantage of the
    shared-namespace cabability when it becomes available.

    1.3.2 Execution (threads and synchronization)

    The main thread of control on each node is that which reads commands
    from the script file (or keyboard if the session is interactive).  The
    parallel library provides limited capabilities for this thread to
    create new threads on any node.  On each node the threads are pushed
    onto a stack with the main thread at the bottom of the stack.  Only
    the topmost thread may execute and when it completes it is popped off
    the stack so that the next thread down can continue.  Threads ready to
    execute are NOT guaranteed to execute: if the topmost thread is
    blocked or looping, no ready thread lower on the stack can continue.

    An executing thread is guaranteed to run to completion (assuming it
    does not contain an infinite loop or block on I/O) so long as it
    executes only local operations, i.e. no operations that explicitly or
    implicitly involve communication with other nodes.  The commands
    descriptions below include specification of local or non-local status.
    In addition, simulation steps and reset are by definition non-local
    operations if there is more than one node in the zone.  Users are
    strongly encouraged to use only local operations in child threads
    whenever possible.  Users need to be very careful about thread
    creation to ensure that deadlock (no thread can continue) does not
    occur.

    The parallel library provides facilites for blocking and non-blocking
    thread creation, usually used to execute commands on nodes different
    from the one the script is being executed on ("remote" nodes).  When a
    thread (including the main script) initiates a blocking remote thread
    (also known as remote procedure call or remote function call), it
    waits until the thread completes before continuing.  When a thread
    initiates a non-blocking remote thread (an asynchronous thread), it
    continues immediately without waiting for termination of the thread.
    While a thread is waiting, the node can accept a request for thread
    creation arriving from any node (including itself).  This new thread
    is pushed on the thread stack and executed, so that the original
    waiting thread does not continue until the new thread has completed.

    Scripts running on different nodes can synchronize via several
    different synchronization primitives.  There are two types of barrier
    (each script waits at a barrier until all have reached it), one which
    involves all nodes in a zone, the other involving all nodes in the
    parallel platform.  By default there is an implicit zone-wide barrier
    before a simulation step is executed, although this can be disabled.
    Pairwise synchronization of nodes is also possible.

    When a script requests that a command be run asynchronously on another
    node it initiates a child thread of control on the other node.  The
    child thread runs asynchronously with its parent.  The parent can
    request notification or the child's result when the child completes,
    and can wait on that notification or result (a "future").  By default
    all threads block for child completion before each simulation step
    although this feature is modifiable by the user.  It is easy to reach
    deadlock (no thread able to continue) if the creation and execution of
    threads is handled carelessly.

    If a node initiates several child threads on a particular remote node,
    these are guaranteed to commence (but not necessarily complete)
    execution in the order in which they were initiated.  A thread is
    guaranteed to execute eventually so long as no preceding thread 1)
    enters a loop which only executes local operations, or 2) blocks
    indefinitely because of deadlock.  Once execution of a thread begins,
    it runs to completion without interruption so long as it only executes
    local operations.

    1.4 Simulation and scheduling

    The parallel library provides the ability to set up a Genesis message
    between to objects on different nodes (a remote message), provided the
    nodes are in the same zone.  Data is physically transferred from one
    node to the next at the beginning of a simulation step (see section
    ??? for user control of this process).  This means that there is no
    transfer of data between objects within a single timestep, which has
    ramifications for the schedule.  The parallel library guarantees that
    execution on a parallel platform will be identical to that on a single
    processor if and only if there are no remote messages for which the
    source object precedes the destination object in the schedule (we
    assume that every node in a zone has the same schedule).

    1.5 Deferred features

    [Mail ngoddard@psc.edu if you would like to see any of these features
    implemented.]

    Features which the parallel library will eventually include but which
    are not currently implemented:

    	remote execution on arbitrary sets of nodes

    	remote messages deletion

    	copy or deletion of elements sending or receiving remote messages

    	inspection of remote messages (e.g., showmsg, getmsg)

    	active messages which have slot data

    	rget (get value of remote script variable)

    	rpoll (wait until remote script variable is non-zero)

    	rvolumeconnect (like volumeconnect)

    =============================================================================
    2.0 Commands

    2.1 Syntax.

    In the interests of backwards compatibility, the syntax of most
    commands is an extension of existing command forms.  The general idea
    is to precede the existing command name with an "r" (for "remote") and
    to add the specification of the remote node with an @.

    	Local syntax (existing) :
    	command args ....
    	element_path

    	Remote syntax
    	command@ args ...
    	element_path@node

    	 <- ,,...

    	 <- [.]

    where  is the znode number for the node, that is its number
    within its zone.  If . is omitted, the zone of the caller is
    assumed.  Any command or element without the @ extension is assumed to
    be on the local node. If the node and zone are the same as the
    caller's, the command is executed locally.

    Nodes and zones are identified at present as an integer (znode number
    or zone number) or one of two keywords:

    other	 :	All except the caller's ("others" also accepted).
    all	 :	All including the caller's.

    Thus "@other" means all nodes in the caller's zone except the caller's
    node.  "@other.all" means every node in the parallel platform except
    the caller's.  "other.other" means every node in every other zone
    except the nodes whose znode number is the same as the callers znode
    number.  "@all.all" means every node in the parallel platform.  "@all.other"
    means every node in every other zone.  If "other" is given for node then
    there must be a node whose znode number corresponds to the caller's
    znode number in every zone that is specified.  There must exist at
    least one node in each node-spec.

    [Deferred: appending "only" to the @ means that the reference is
    exclusively to those nodes specified in the node-list - all other
    nodes do nothing. Thus "command@only-2,4,6" results in the command
    being executed on nodes 2,4 and 6 only.  Mail ngoddard@psc.edu if
    you would like this implemented.]

    =============================================================================
    2.2 Startup

    	paron
    	syntax : paron [option [arguments]]+

    	This command initializes the parallel simulation It performs the
    	following tasks :

    	* Creates the postmaster /post
    	* If global node zero then spawn all the other nodes
    	* Partitions the parallel platform into zones
    	* Adds the parallel event loop ParEventLoop as a job at priority 1
    		(which is high but not top)

    	The command should be issued before starting any parallel functions.

    	2.2.1 paron syntax options.

    	Several paron options control how the nodes are spawned by the master.
    	Worker nodes ignore these options in the paron command, except the
    	-debug option.

    	-executable filename	(default: "parnxgenesis")
    		the file in $HOME/pvm4/bin/ARCH to execute

    	-nodes number		(default: $NNODES if it is defined, otherwise
    					  the size of the parallel machine)
    		the number of nodes to use in the parallel platform

    	-execdir directory-name	(master's working directory at paron time)
    		the name of the directory which will become "." for the workers

    	-altsimrc filename		(default: ".parnxsimrc")
    		the Genesis configuration file to use, located in $HOME or "."

    	-startup filename	(default: same name as master's startup file)
    		the name of the .g file on the SIMPATH to read on startup

    	-silent level		(default: 3)
    		the Genesis level of error/feedback reporting

    	-debug flags		(default: 0)
    		the PVM debug options for spawning tasks.  See the PVM
    		manual and section ??? for details.  4 means debug.

    	-output filename	(no default)
    		the name of the file (relative to the master's working
    		directory if not an absolute pathname) to which
    		stdout/err from workers is directed

    	-nice level		(master's nice level)
    		the nice level to run the workers at.  Positive values only,
    		a higher level means a lower priority.

    	-dbgout filename	(stdout, for implementors only)
    		the name of the file which workers will print debug messages
    		to, an extension is added (the worker's PID).
    		[NOT IMPLEMENTED]
    	-batch
    		if present, the worker will be run in batch mode
    		so any interaction with the user will be suppressed

    	The remaining options for paron relate to specification of zones.
    	It is an error, currently NOT detected, if the nodes do not all
    	have the same specification of zones.

            -farm
    	        each node is in a separate zone (default).

            -parallel
    		all nodes in the same zone.

    	-mixed nnodes nzones [nnodes nzones] ...
    		This is the most general option. It sets up 'nzones' zones
    		with 'nnodes' nodes in each. Out of the remainder, a similar
    		specification can be used, and so on. The nodes left over
    		are placed in single-node zones.  If
                    nnodes or nzones is <1 it means divide the remaining
                    number of nodes by the non-negative value to obtain
                    the other value. Remainders go on as before.
    			There is a special case when the first zone is of
    		size 1 (e.g. paron mixed 1 1	 10 10 ....). This case
    		usually occurs when one wants a master node to coordinate
    		the operation of all the rest, as in a parameter search
    		simulation. In this situation the zero zone is placed
    		at the real zero node of the parallel machine, which is
    		usually the processor with a TTY interface if any exiss.

    	2.2.2	Examples
    	paron
    		By default this just sets up a simulation where each node is
    		in a separate zone.

    	paron -parallel
    		Sets up a simulation where each node is in the same zone, and
    		can send messages to each other.

    	paron -mixed 10 5     3 others
    		If this were starting with 64 nodes, the simulation would look
    		like this :
    		Zone 0,1,2,3,4   : 10 nodes each  = 50 nodes
    		Zone 5,6,7,8     : 3 nodes each   = 12 nodes
    		Zone 9,10        : 1 nodes each  =  2 nodes

    -----------------------------------------------------------------------------
    2.3 Local commands
    -----------------------------------------------------------------------------

    	mynode,nnodes,myzone,nzones,ntotalnodes,mytotalnode

    	mynode - number of this node in this zone
    	nnodes - number of nodes in this zone
    	myzone - number of this node's zone
    	nzones - number of zones
    	ntotalnodes - number of nodes in all zones
    	mytotalnode - unique number over all zones for this node

    	These are utility functions for finding out about
    	configuration of the parallel platform.

    	Syntax : They return an int, and do not take any
    	arguments.

    	Examples :
    		echo {mynode} {nnodes} {myzone} {nzones}

    -----------------------------------------------------------------------------
    2.4 PVM access
    -----------------------------------------------------------------------------

    		mypvmid, npvmcpu

    		mypvmid - task identifier used by PVM for this node
    		npvmcpu - number of cpus used by PVM in the parallel machine

    		These functions take no arguments and return an integer.
    		They give access to some of the underlying PVM information.

    -----------------------------------------------------------------------------
    3.4	threadson, threadsoff, clearthreads

    This set of functions is used for controlling execution of threads.
    The paron command leaves the parallel library in a state in which
    suspended threads ready to resume and threads launched by a remote node
    can be executed when the current thread suspends itself.  The threadsoff
    command disables the (re)commencement of any other thread.  The threadson
    command enables the (re)commencement of other threads.  The clearthreads
    command executes all ready threads until completion or until they suspend
    themselves (and even these may complete if they become ready again before
    the queue of ready threads becomes empty).

    Syntax : No arguments, no function values returned.

    3.4.1	threadson : This operation re-enables parallelism.

    3.4.2	threadsoff : This operation disables parallelism. Useful
    	if one is performing a function (eg, simulation steps)
    	which should not be interrupted to perform script functions.
    	Also useful if one wishes prevent access (via rpc calls) to script
    	variables till they have been evaluated.

    3.4.3	clearthreads : This operation clears all pending parallel setup
    	commands. It is meant to be used after threadsoff has been
    	called, to complete execution of any remaining threads.

    -----------------------------------------------------------------------------
    Non-local commands
    -----------------------------------------------------------------------------

    3.3	remote procedure call

    3.3.1	Syntax:
    		command@node-list args [-reduce type]
    		node-list <- node-spec[,node-spec]...
    		node-spec <- node[.zone]

    NOTE: until a parser bug is fixed, use this variant if any node-specs
          are enclosed in {} for evaluation by the parser, e.g. func@{i}.
          The resulting warning message that can be ignored:

    		rcommand command@node-list args [-reduce type]

    This causes a command to be executed on one or more nodes.  It
    suspends the script, creates a new thread on the nodes (usually remote
    nodes), executes the command, and resumes the script when the command
    has completed on all nodes.  While the script is suspended other
    threads may execute (other suspended threads which resume, or threads
    created by remote calls from other nodes).  The result of the command
    is returned as a character string - note that this result is undefined
    if the function is executed on more than one node.

    [Deferred: If the the optional -reduce flag is given, the appropriate
    reduction operation is performed on the results from all the nodes on
    which the command is executed.  The result after reduction is returned
    as a character string.  Mail ngoddard@psc.edu if you would like this
    implemented.]

    A heuristic to use to avoid deadlock in a remote procedure call is to
    be sure that the function that is executed only performs local
    operations.  That is, it should not issue any of the remote operations
    listed in this document, nor execute any simulation steps for a zone
    containing multiple nodes.  Local operations are guaranteed to execute
    without suspending the thread, so that the remote thread which
    implements the rpc call is guaranteed to execute to completion once it
    begins execution.  Violating this heuristic can easly lead to
    livelock, deadlock, or infinite loops.

    Example :
    	On node 0 we set up the function
    		function foo(x)
    			int x
    			return({x*10})
    		end

    	On node 1 we issue the command

    		int myfoo
    		myfoo = {foo@0 2}
    		echo {myfoo}

    	and the reply is :

    		20

    -----------------------------------------------------------------------------

    3.3.1	async
    	Syntax:
    		async function[@node-list] args... [[-complete]|[-scalar]]

    		node-list <- node-spec[,node-spec]...
    		node-spec <- node[.zone]

    The async command causes one or more new threads to be created to
    execute the function on the node or nodes specified in the nodelist.
    These threads execute asynchronously with the script.  The function
    can be a Genesis shell command (including Unix shell commands) or a
    script function.  The async command returns as soon as the request for
    new threads has been sent.  If the [-complete] or [-scalar] flag is
    given, the script can synchronize with the completion of the
    asynchronous thread(s), and obtain a result in the case of -scalar,
    using the "waiton" command.  If no node is specified, the current node
    is assumed.  The async command has special meaning when applied to the
    raddmsg command, see the documentation on raddmsg.

    A heuristic to use to ensure that the remote thread completes is the
    same as that for rpc callse, namely to be sure that the function that
    is executed only performs local operations.  That is, it should not
    issue any of the remote operations listed in this document, nor
    execute any simulation steps for a zone containing multiple nodes.
    Local operations are guaranteed to execute without suspending the
    thread, so that the remote thread which implements the rpc call is
    guaranteed to execute to completion once it begins execution.
    Violating this heuristic can easly lead to livelock, deadlock, or
    infinite loops.

    		Examples :

    		async echo@2 foo
    			will cause node 2 in the caller's zone to echo 'foo'.

    		async echo@all foo
    			will cause all nodes in the caller's zone, including
    			the current one, to echo "foo".

    		async echo@all.others {mynode}
    			will cause all nodes in all zones except the zone of
    			the caller to echo the node number of the caller.
    			Note that the curly braces are evaluated on the
    			issuing node, not at the destinations !

    		async@1 echo@2 foo
    			will tell node 1 to asynchronously tell node 2 to
    			echo foo. This is an example of a remote thread
    			call that violates the heuristic for ensuring the
    			thread completes.

    -----------------------------------------------------------------------------
    3.3.2	waiton

    	Syntax:
    		waiton

    This command is used in conjunction with the "async" command.  If async is
    given the -complete or -scalar flag, it returns an integer (a "future")
    which can be used as an argument to the "waiton" command.  The effect is
    for the script to suspend until the command that was started asynchronously
    has completed.  If the -scalar flag was given, "waiton" returns the result
    of the command as a character string.  This implements a form of "future":
    the value returned by "async" is a promise of a future value which is retrieved
    with "waiton".  If the -complete flag was given, then a form of split-phase
    computation is possible.  The "async" command is used to initiate a computation
    and the "waiton" command used to wait for it to complete, with the script able
    to perform useful work that does not depend on completion between the two.

    	Examples:

    		// using a future to get a result
    		int future, result
    		// run command asynchronously putting the result in a future
    		future = {async command@node args... -scalar}
    		// do some useful work
    		...
    		// get the result from the future
    		result = {waiton future}

    		// using a future for split-phase computation
    		int future
    		// run command asynchronously returning a future
    		future = {async command@node args... -complete}
    		// do some useful work
    		...
    		// synchronize with the asynchronous command's completion
    		waiton future

    -----------------------------------------------------------------------------
    -----------------------------------------------------------------------------

    3.5	barrier, barrierall

    These functions synchronize nodes.  A node waits at a barrier until
    all the other nodes have reached it.  The command must be
    issued for all nodes, and when encountered by any node it
    blocks till all other nodes have also encountered it.

    		Syntax :
    		barrier [barrier_id [timeout_seconds]]
    		barrierall [barrier_id [timeout_seconds]]

    Every barrier has an id number, if no number is given the default is
    specified.  Barriers only match if the barrier id matches.  Valid id
    numbers are from 1 to 24.  Providing barrier ids in a parallel script
    helps to verify program correctness (by causing deadlock for incorrect
    programs).  If no barrier ids are provided, then any barrier will
    match any other barrier in the script.  A timeout in seconds can be
    provided to barrier commands, otherwise the default timeout value is
    used.

    3.5.1	barrier : This applies to all nodes within a zone. The call
    	blocks till all nodes in the zone have called barrier with the
    	given barrier id.

    3.5.2	barrierall : This applies to all nodes in all zones in the
    	simulation. The call blocks till all nodes in all zones
    	have called barrierall with the given barrierall id.

    -----------------------------------------------------------------------------
    3.7	raddmsg
    	This function sets up a message between elements on different nodes
    	but in the same zone.

    	Syntax :
    	raddmsg src dest[@node] MSGTYPE var1 var2..

    	(Note that this is the same as for addmsg.)

    As with addmsg, the simulation must be reset before the messages take
    effect properly.  Only the variable fields will be transmitted
    on every time step. The constants will be set up by the command
    and then stay fixed.

    EFFICIENCY NOTE: The raddmsg command requires some communication
    between the source and destination nodes before it can complete, and
    the command blocks while this is occurring.  If raddmsg is executed
    asynchronously using the "async" command, then two efficiencies result
    which can greatly speedup setup phase of a script.  First, the script
    need not wait for a remote message to be completed before starting on
    the next remote message.  Secondly, raddmsg recognizes that it is
    being called asynchronously and optimizes the communication of message
    specfication between nodes.  In this case, remote message
    specifications are buffered on the source node until there are a large
    number for a given destination node.  These specifications are shipped
    off in one large communication to the destination node, which sends
    back one large response.  Buffers are flushed (i.e., remote messages
    completed) whenever a barrier or step command is issued or whenever
    required by a "waiton" command.  Remote messages from the source node
    to a given destination node are guaranteed to be completed in the
    order they are issued in the script.

    Examples:

    	// connect vector of elements to corresponding elements on other nodes
    	// do it asynchronously
    	for (i = 1; i < n; i = i+1)
    	  async raddmsg elt[i] elt[i]@others ...
    	// guarantee all have completed with a synchronous call
    	raddmsg elt[0] elt[0]@others ....

    Bugs:	Messages created in this way cannot be deleted yet.
    	These messages are not reported properly by showmsg or
    	getmsg yet.
    	Active messages cannot have slots yet.
    	Constant fields are not yet implemented.

    -----------------------------------------------------------------------------
    3.8	rvolumeconnect [DEFERRED]

    	Just like volumeconnect except the destination element specification
    	can include a node-list (e.g., "@all").  When implemented this will
    	be a much more efficient way to make multiple, spatially defined
    	connections.

    -----------------------------------------------------------------------------

    3.10	rget [DEFERRED]

    	This function returns the value of a global script variable
    	on a remote node. It can only be called for a single remote
    	node. It returns a string. It is a blocking call, which
    	will not mean much unless the remote node has its threads handler
    	turned off.

    	Syntax :
    	rget global[@node[.zone]]

    Example :
    	(on node 1.1) :
    		str foo="hello there"

    	(on another node)
    		echo {rget(foo@1.1)}
    			hello there

    -----------------------------------------------------------------------------

    3.11	waitset  [DEFERRED]

    	This function polls the value of a global script variable
    	on a remote node, and blocks until that value is nonzero. It
    	returns the value of the remote global when it finally comes
    	through.  The function returns a string. However, for remote
    	string variables it will not return unless the string is a
    	nonzero number.

    	Syntax :
    	waitset global[@node[.zone]] [time [hang_time]]

    	The time option allows one to set the polling interval.
    	It defaults to 1 sec.

    	The hang_time option allows one to tell the function when
    	to give up. It defaults to the postmaster hang_time.

    	Example :
    	(on node 1.1) :
    		int foo=0

    	(on node 0.0)
    		echo {waitset(foo@1.1)}
    	(the command blocks...)

    	(on node 1.1) :
    		foo=123

    	(on node 0.0, the poll command finally returns)
    			123
    -----------------------------------------------------------------------------