Technical Details and Initial Setup on kali

This page can be reached via kali's webpage at http://www.math.umbc.edu/kali.

Purpose of this Document

This document collects some additional technical details about kali and software installed on kali, in addition to the information on kali's main webpage. Moreover, this document explains how to compile code for parallel computing using MPI on kali. Additionally, it explains how to get set up to use MPI and collects some additional information that might be useful upon your first login. While you generally need to read this information only once, you may want to come back to it if you see unexplained performance of kali that this information might explain.

If you find mistakes on this page or have suggestions, please contact me.

More Details on the Hardware and the Networks

Let me start by explaining the setup of the cluster again: It consists of 1 head node (full hostname kali.cl.math.umbc.edu or for short kali) (a combined management and user node) and 32 computational node; the latter are subdivided into 1 storage node (storage1), and 31 compute nodes (node01 through node31). All machines have two Intel Xeon 2.0 GHz CPUs with 512 kB cache. The compute nodes have 1 GB of memory, while the head and storage nodes have 4 GB each. The 32 computional nodes are connected via a high-performance Myrinet interconnect, thus making 64 processors available for computations.

Only the head node can be accessed from the outside of the machine; it communicates to all other nodes via the cluster network comprising a 100 Mbps ethernet network. The main file storage is on a RAID array on the storage node, which is NFS-mounted to the head node and all other nodes via the cluster network. This RAID array consists of 8 SCSI harddrives with 1 hot swap and striped check-sums as backup; this setup allows for the failure of up to 2 disks without having to retrieve data from backup tapes.

This network schematic illustrates visually the network connectivities of the Myrinet and ethernet networks, the connection to the outside world, and the placment of the RAID array; notice how the connection speeds are marked, pointing out that the ethernet switch is a special one with two faster 1 Gbps ports to prevent the two most-often used connections, to the management and storage nodes, from becoming bottlenecks. A third network, the management network, is not shown.

Finally, after the abstract information given so far, the rack schematic shows how the components discussed are arranged in the two racks. The labels H5 and H6 denote the placement of the racks on the floor of the operations room of UMBC's Office of Information Technology, where the machine is housed. This schematic might be useful to know, when looking at the "Physical View" in Ganglia. (Notice that the nodes have been renamed from the form node0XY with three numerical digits to the form nodeXY with two numerical digits during the operating system upgrade in March 2005.)

Some photographs of kali:

The Partitions on the RAID Array

The RAID (= redundant array of independent disks) array connected to the storage node storage1 constitutes the main user disk space on kali. It is nominally a 0.5 TB array, but due to redundancies for data safety and a spool partition for the tape backup system, about 367 GB are the usable space.

The RAID array is divided into two partitions, one for users' home directories (/home, 100 GB), which is backed up onto tape, and a scratch partition (/scratch, 267 GB) that is never backed up. Users should do their calculations in the /scratch partition and copy relevant results to the home directory (or other computers than kali) for safe-keeping. There is a quota of 1 GB that applies to your part of the /home partition.

The /scratch partition is accessed by changing directory to /scratch/username, where username is your username on kali. For your convenience, a symbolic link called scratch in your home directory links to this directory directly; it appears and is used just as if it were a real directory, i.e., you can say cd scratch in your home directory to get to /scratch/username.

How to Login to kali

The only machine with a connection to the outside network is the head node; internally to the system, its full hostname is kali.cl.math.umbc.edu, but from the outside, it appears as kali.math.umbc.edu. To log in to the system, you must use a secure shell like ssh. Therefore, you can use ssh kali from inside the department (with identical username) or more generally ssh -l username kali.math.umbc.edu or equivalently 
  ssh username@kali.math.umbc.edu
from anywhere on the internet, where username denotes your username on kali. Your username and password will be the same as on other campus systems such as the gl system. See man ssh for more general information.

When you connect to a machine for the first time with ssh, your own machine should warn you about this fact and ask for confirmation that you want to connect. If you use ssh routinely, you will be used to this. I wanted to point out that, additionally, the ssh on kali will perform some setup that allows you to connect to the nodes of kali without typing in passwords. You will see lines such as 

  generating ssh file /home/gobbert/.ssh/id_dsa ...
  Generating public/private dsa key pair.
in this process.

How to Copy Files To and From kali

This cluster only allows secure connection from the outside. Hence, the above instructions referred to ssh. Correspondingly, to transfer files in and out of the machine, you must use scp or similar secure software. You will need to do this to copy results or code to another computer, for instance.

Let me explain the use of scp by the following example: The user username has a file hello.c in sub-directory math627/hw1 from his home directory on kali. To copy the file to the current directory on another Unix/Linux system with scp, use 

  scp username@kali.math.umbc.edu:math627/hw1/hello.c .
Notice carefully the period "." at the end of the above sample command; it signifies that you want the file copied to your current directory (without changing the name of the file).

As with ssh, you can leave out the username@, if your username agrees on both machines. If issuing the command from within the Mathematics domain at UMBC, you can also abbreviate the machine name to kali. See man scp for more information.

The Available Compilers and MPI Implementations on kali

To take advantage of the high-performance Myrinet interconnect on this system - the whole rationale for its use, after all! -, you must use the implementation of MPI provided by Myricom, the maker of Myrinet. This is a modified version of the popular mpich implementation of MPI.

To specify the programming environment completely, you still have to select which compiler to use; you must then use the MPI installation compiled against the selected compiler. Two suites of compilers are available on kali, one from Intel and one from the GNU. Specifically, the Intel compilers are icc for C/C++ and ifort for Fortran 90/95. And the GNU compilers are gcc for C/C++ and g77 for Fortran 77.

The corresponding MPI implementations are in the /usr/opt/mpich-1.2.5.9-ch_gm-icc directory tree for the Intel compilers and under /usr/opt/mpich-1.2.5.9-ch_gm-gcc for the GNU compilers. Notice that you should always compile your MPI code using the MPI scripts for your language such as mpicc for C; see below for more information. The names of the serial compilers above is only provided as background information.

As a matter of background, I mention that for successful compiling, linking, and running, several things have to come together correctly: Your path has to include the location of the correct compile script, the MPI header files have to be found, and the correct MPI libraries have to be found. This is all conveniently set up correctly for you by the switcher utility. See its man page for more information (man switcher), but the most needed commands are these:

Compiling and Linking MPI Code on kali

Let's assume that you have a C code sample.c that contains some MPI commands. The compilation and linking of your code should work just like on other Linux clusters using the mpich implementation of MPI. Hence, compile and link the code both in one step by 
  mpicc -o sample sample.c
The script mpicc works on the surface just like any regular compiler. For instance, the option -o sample chooses the name of the output file (here the executable file), and mpicc compiles and links in (seemingly) one step.

If your code includes mathematical functions (like exp, cos, etc.), you need to link to the mathematics library libm.so. This is done, just like for serial compiling, by adding -lm to the end of your combined compile and link command, that is, 

  mpicc -o sample sample.c -lm
In a similar fashion, other libraries can be linked.

More formally, I could also separate the compile from the link step. That is, the C file sample.c is first compiled into object code sample.o (default output name, if the option -c is used), which then gets linked to the required libraries to obtain the executable sample. The sequence of the two commands 

  mpicc -c sample.c
  mpicc -o sample sample.o -lm
accomplishes this, where the option -c stands for "compile-only".

See the man page of mpicc for more information by saying 

  man mpicc

If you want to compile C++ or Fortran programs, the logic of the commands is the same as above. Simply replace mpicc by mpiCC, mpif77, or mpif90. See their man pages for additional information.

Notice that the MPI compile scripts mpicc, mpiCC, mpif77, and mpif90 have the same name for each of the MPI implementations available. Which one is used depends on which implementation you have selected with switcher. Notice also from their man pages that any option that is not recognized as a MPI option is passed through to the underlying compiler; so you might want to know the name of the underlying compiler to be able to pull up its man page. Caution: As you saw from the above list, GNU does not have a Fortran 90 compiler, so it is not clear what mpif90 might do when using the GNU MPI implementation; let me know if you find anything out about this and other issues, so I can post more details here.

Finally, to be doubly sure which compiler is accessed by your MPI compile script, you can use the -show option as in 

  mpicc -o sample sample.c -show
This option causes mpicc to show what compile commands it would issue without actually issuing them; then notice at the beginning of every line what compiler you see.

Copyright © 2003-2005 by Matthias K. Gobbert. All Rights Reserved.
This page version 3.6, June 2005.