Running parallel programs on an AP1000+ running APLinux is quite straight forward. The basic utilies are prun, paralleld, apmon and the aplib and MPI communications libraries.
prun is used to launch parallel processes, thereby allowing standard pre-linked utilities to be run in parallel as well as message passing parallel applications.
The paralleld daemon runs on each cell, communicating with each instance of prun and generally controlling the execution of each task assigned to its cell.
The apmon program provides a monitoring and control window over the state of the AP+, and a direct communication mechanism with other users - a feature which facilitates the sharing of the machine in a very direct and civilised manner, obviating the need for strict scheduling mechanisms in interactive sessions. This, combined with the ability to run many applications of arbitrary configurations at the same time, makes the system very versatile.
The architecture of the ap1000+ and APLinux are decribed elsewhere, and should be surfed first. The salient features, however, are that:
Parallel programs are launched via prun, the simplest form of which is
% prun progname
run on hosts visible to the AP+ subnet, including AP+ nodes. It is usually installed in /usr/local/bin. Running prun with no arguments will supply a usage clause,
progname does not have to be a parallel program linked with a message passing library. For example, you could run the standard utility hostname on each cell by
% prun /bin/hostname
prun combines the standard output of each nodes invocation into its own standard output, which may of course be redirected. Standard input to prun may be redirected to each cell by specifying the -s switch.
I find the most useful way of capturing lots standard output is through the usr of the -WL out switch, which will produce a separate file of each cells output in out.N for cell N
It is possible to run a parallel program over any combination of cells in any size n x m mesh as long as the total number of cells is not greater than then number of physical cells available. The simplest way to do this is with the -n switch (see the usage), but can also be set by the environment variables CAP_NCEL, CAP_NCELX, CAP_NCELY and CAP_NODE_LIST. Further details can be found in the section on Controlling the Execution Environment.
| Usage: | prun [options] program |
| prun [ -c | -p ] |
| -n | num | : specify num nodes (or) |
| -n | xxy | : specify x by y mesh |
| -s | : enable standard input to all cells | |
| -WF | file | : enable CellOS style logging to file |
| -WL | file | : enable logging to file.XX |
| -WS | : enable logging to stdout/stderr | |
| -c | : print cell config | |
| -p | : probe socket tree |
It is possible to create tightly coupled parallel applications under APLinux by linking with the aplib message passing library. There is also an MPI library implemented for the AP+, which is described in detail elsewhere.
The interface to the standard aplib calls is identical to the CellOS AP1000 library, but the library has been substantially rewritten to allow multiple applications to run simultaneously under a linux environment. aplib applications can also run over arbitrary mesh configurations and logical to physical cell mappings. The library utilises the MSC+ message passing hardware directly from the users context, resulting in efficient, low latency performance.
Consult your local system documentation for the intallation details of aplib.
You will need to determine a path for aplib itself, and the location of
the standard aplib header, ccell.c7.h.
Usually, aplib is installed in /hostname/usr/local/lib, for the
AP1000+ system called hostname.
It should also be visible
on the cells in /usr/local/lib, which may be optimal on machines where this
filesystem is mounted on local disks, when using the shared library.
Similarly, ccell.c7.h should be in
/hostname/usr/include/aplib.
Some limitations are that currently there is no facility for creating host programs, and all applications run similarly to the HAP (Host Access Package) CellOS environment. There is only a C language interface, with a Fortran version being developed.
There is a brief overview of the aplib interface in postscript.
The AP+ system monitor, apmon, is a very useful and easy to use console device for anyone using APLinux on the AP+. The current status of all cells can be monitored, along with the status of parallel jobs and other users. It can also be used to communicate with other users, control processes and reboot the system.
apmon is installed in /usr/local/bin, and can be run continuously in the foreground by entering apmon in a spare xterm. Help for general apmon commands can be obtained by entering a '?' or typing help at the apmon non-prompt. More explicit kernel based commands are shown by entering a 'k' to apmon.
A useful example is the 'P' command, which shows the currently running parallel applications on the cell being monitored. Monitoring all cells is the default, but can be altered using the 'cell' command. cell -1 specifies the default.
Each cell runs the paralleld daemon, which are all connected via a socket tree to the root daemon. paralleld is responsible for launching all parallel programs and multiplexing stdio streams.
When a program is launched by prun, a socket connection is established to the root paralleld, the image is uploaded to the appropriate cells. When all cells invocations are complete, prun is notified and terminates.
paralleld is now fairly robust, but if your program fails to terminate, or to produce expected output, it may be because the paralleld on a cell may have died or lost its socket connection. This can be tested by prun using the -p (probe) option. If there are less responses than physical cells, then contact the AP+ system administrator. Otherwise, typical problems are; shared objects not being visible over NFS, long path variables placing a high demand on NFS, and inappropriate permissions on mounted filesystems.
The default logical configuration of the AP+ under APLinux is identical to the physical configuration. Other configurations can be specified by the -n switch in prun as explained in the Overview, or by setting environment variables. This method facilitates batch run loops over varying sized meshes quite easily, and can also force particular cells to participate in the mesh in the case of a heterogeneously resourced system - while maintaining the logical mesh structure expected by the aplib calls (including broadcasts and the xy_* calls).
The variables used to control the configuration are:
Running the command
% prun printenv | grep CAP
with various variable settings should clarify the semantics of the possible combinations. You should notice the variable CAP_NODE_LIST in the output. This variable specifies the actual physical cells participating in the configuration, and is set to optimal values for broadcast and global reduction operations if unset.
It can be set, however, by specifying a colon separated list if integers in the range 0 .. (Physical_cells - 1). For example, using a csh:
% setenv CAP_NODE_LIST 4:8:9:1
will ensure that these physical cells will be the lowest ordered logical cells in the configuration. If the configuration is larger than the number of elements in CAP_NODE_LIST, then the list is filled in with unused nodes, depending on available system resources