%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 
\chapter{Parallel Computing and Beowulf Clusters}\label{chapter:parallel}

Parallel Computing is a strategy for performing large, complex computational 
tasks
faster. Large tasks can either be performed serially, one step
following another, or if possible can be decomposed into smaller tasks to be
performed simultaneously---that is, in parallel. Thus, the terminology
``parallel computing'' refers to the use of multiple computers or
processors working together on a common task. 

\section{Taxonomy of Parallel Computer Architectures}
Currently, the most popular nomenclature for the classification of
parallel computer architectures is that proposed by Flynn~\cite{flynn}
in 1966 (see Fig.~\ref{fig:flynn}). Flynn chose not to examine the
explicit structure of the machine, but rather how instructions and data
flow through it. Specifically, the taxonomy identifies whether there are
single or multiple ``streams'' for data and for instructions. The term
stream refers to a sequence of either instructions or data operated on
by the computer. Single instruction refers to the fact that there is
only one instruction stream being acted on by the Central Processing
Unit (CPU) during any one clock tick; single data means, analogously,
that only one data stream is being employed as input during any one
clock tick. The four possible categories of Flynn's taxonomy are:
\myfigureW{figs/flynn.ps}{flynn}{10cm}
{Flynn's Classification}
\begin{enumerate}
\item {\bf Single Instruction Single Data (SISD)}:
SISD machines are conventional serial computers that process only one
stream of instructions and one stream of data. A SISD machine
is equivalent to a von Neumann machine.
Instructions are executed sequentially but may be overlapped by 
pipelining---most SISD systems are now pipelined. SISD computers may
have several functional units, examples include extra mathematics
coprocessors, vector units, graphics processors and I/O processors. As
long as there is only one CPU the architecture remains SISD. Examples of
typical SISD machines are the widely-used single-processor desktop PCs
and workstations.

\item {\bf Single Instruction Multiple Data (SIMD)}:
This category encompasses computers which have many identical 
interconnected processors under the supervision of a single control
unit. The control unit transmits the same instruction,
simultaneously, to all processors. The topology of the communications
network is not addressed by Flynn's taxonomy.
All the processing elements simultaneously execute the same instruction
and are said to be ``lock-stepped'' together. Each processor works on data
from its own memory and hence on distinct data streams. Every processor
must be allowed to complete its instruction before the next instruction
is taken for execution. Thus, the execution of instructions is said to
be synchronous. Examples of SIMD machines are the ILLIAC-IV, Connection
Machine (CM-1 and CM-2) and MasPar.

\item {\bf Multiple Instruction Single Data (MISD)}:
MISD machines have many processing elements, all of which execute 
independent streams of instructions. However, all the processing 
elements work on the same data stream. There are two ways in which this
can be done: firstly the same data item can be fed to many processing
elements each executing their own stream of instructions.
Alternatively, the first processing element could pass on its
results to the second processing element and so on, thus forming a
macro-pipeline.
The only known example of a computer capable of MISD operation is 
the C.mmp~\cite{cmmp} built by Carnegie-Mellon University, which
is reconfigurable and can also operate in SIMD and MIMD modes.

\item {\bf Multiple Instruction Multiple Data (MIMD)}:
Most multiprocessor systems and multiple computer systems can be placed
in this category. A MIMD computer has many interconnected processing
elements, each of which have their own control unit. The processors
work on their own data with their own instructions. Tasks executed by
different processors can start or finish at different times. They are
not lock-stepped, as in SIMD computers, but run asynchronously. Examples
of MIMD machines are Intel Paragon, IBM SP2, Cray C90, and  clusters of
workstations running programs based on Parallel Virtual Machine (PVM) or
Message Passing Interface (MPI).

MIMD systems can be further divided coarsely based on memory organization: 
\myfigure{figs/parallel.ps}{mimd}{7.5cm}{15cm}
{Classification of Parallel Computers}
\begin{enumerate}
\item {\bf Shared Memory}:
    Shared-memory systems can be divided into the following three sub-categories. 
\begin{enumerate}
\item {\bf Symmetric Multi-Processing (SMP)}: 
In SMP systems,
the memory is physically shared, and all processors access the memory equally at equal speeds. 
\item {\bf Non-uniform Memory Access (NUMA)}: 
In NUMA systems, the memory is physically shared, but not distributed in a 1:1 relation with the processors. Due to the non-uniform distribution, the access to different portions of the memory may require significantly different times. 
\item {\bf Distributed Shared Memory (DSM)}: In DSM systems, the memory is distributed among the processors, but the system gives the illusion that it is shared. 
\end{enumerate}
\item {\bf Distributed Memory}:
    Distributed memory systems can be divided into the following three sub-categories based on the network connection between the multiple processors. 
\begin{enumerate}
\item {\bf Fixed}: In fixed distributed memory systems, the number of connections is fixed as more processors are added (e.g., Ethernet-connected workstations, since the Ethernet interconnection is a single resource that all processors share). 
\item {\bf Linear}: In linear distributed memory systems, the number of connections grows linearly with the number of nodes (e.g., mesh-connected multicomputers such as the Intel Paragon). 
\item {\bf Scalable}: In scalable distributed memory systems, the number of connections grows as $P \log P$ or greater where $P$ is the number of processors (e.g., hypercubes such as the Intel iPSC/860). 
\end{enumerate}
\end{enumerate}
\end{enumerate}

\section{Parallelization Strategies}
Parallel computing paradigms fall into two broad classes: {\it
functional decomposition} and {\it domain decomposition}. These
classes reflect how a problem is allocated to processes. Functional
decomposition divides a problem into several distinct tasks that
may be executed in parallel; one field where this is currently popular 
is that of Multidisciplinary Design Optimization (MDO)~\cite{MDO}. 
In MDO, the objective function involves
   numerical simulations from more than one physical discipline which
can be efficiently implemented in parallel. 
On
the other hand, domain decomposition distributes data across processes,
with each process performing more or less the same operations on the
data. Domain decomposition is to be distinguished from the SIMD model:
in the SIMD model, each process executes the same instructions at
the machine level--that is, all processes proceed in lockstep, doing
exactly the same operations on different data. This kind of parallelism
(sometimes called fine-grain parallelism) is usually found at the loop
level, whereas domain decomposition is applied at the problem level.
In the {\it Single Program Multiple Data} (SPMD) parallelism, the same
code is replicated to each process. One process acts as the master and
collects the results from all other processes, thus doing slightly more
work than the other processes. Each processor works on its section
of the problem and is allowed to exchange information (data in local
memory) with other processors.

There are several reasons for the choice of parallel computing over a
conventional single processor computation. Single processor computers
have a limited amount of memory and cannot process data faster than
their rated speed (which is also limited by the current semiconductor
technology). Parallel computing not only allows us to solve problems
that do not fit on a single processor machine, but is also capable of
obtaining the solution in a short amount of time (dependent on the
size of the problem and the number of processors employed). Thus, one can 
run larger problems than were previously possible, and obtain solutions
to these problems much quicker.
Parallelism can be achieved by:
\begin{itemize}
\item breaking up the task into smaller tasks;

\item assigning the smaller tasks to multiple workers to work on
simultaneously; and

\item coordinating the workers.
\end{itemize}

Traditionally, programs that have been written for serial computers
execute only one instruction at a time, use one processor, and
process instructions dependent on how fast data can move through
hardware. The current fastest processors execute approximately $2.5$
billion instructions per second~\cite{spec}. One might think that this
is fast enough, but there are several classes of problems that require
faster processing such as various simulation and modeling problems,
problems dependent on computations/manipulations of large amounts of
data, and grand challenge problems such as climate modeling, fluid
turbulence, human genome, etc. Computers that support parallel computing
are broadly referred to as High Performance Computing (HPC) systems.

\section{Beowulf Clusters}

The Beowulf cluster is a multi-computer MIMD architecture which can be used
for parallel computations. The word ``Beowulf'' was borrowed from one
of the earliest surviving epic poems written in English, which tells
the story about a hero (in this case the ``commodity computer'') of
great strength and courage who defeats a monster (in this case the
``existing supercomputers''). Based on the previously discussed
taxonomy, it is a fixed distributed memory MIMD machine. A typical
Beowulf cluster consists of one server node, and one or more client
nodes connected together via some network. A schematic of a typical
Beowulf cluster is shown in Figure~\ref{fig:beowulf}.

\myfigureW{figs/beowulf.ps}{beowulf}{10cm}
{Schematic of a typical Beowulf cluster}

The first Beowulf cluster was built by NASA in 1994~\cite{beowulf}
and consisted of $16$ 486DX4-100 MHz machines each with $16$ MB of
memory---the best available commodity hardware of the time. Beowulf is
a system built using commodity personal computers, standard ethernet
adapters, and switches. It uses Cost-effective Off-The-Shelf (COTS)
components. It does not contain any custom hardware components and is
trivially reproducible. Beowulf also uses commodity software such as
the Linux operating system~\cite{linux}, PVM and MPI, and other widely
available open-source software. A Beowulf system behaves more like
a single machine rather than many workstations as the server node
controls the client nodes transparently. Today, the price-performance
(or dollars/MFLOPs) for these cheap clusters is far better (often
10 times or more) than for a typical supercomputer from IBM, SGI or
Hitachi, making them extremely attractive for small organizations and
research groups.

\myfigureW{figs/cocoa2.ps}{cocoa2}{6.5cm}
{COst effective COmputing Array-2 (COCOA-2)}

The COst effective COmputing Array (COCOA)~\cite{cocoa} is a Penn
State Department of Aerospace Engineering initiative to bring low cost
parallel computing to the departmental level. COCOA is a $50$-processor
cluster of off-the-shelf PCs connected via fast-ethernet ($100$
Mbit/sec). The PCs are running RedHat Linux version 7.0 with MPI for parallel
programming and DQS~\cite{dqs} for queuing the jobs. Each node of COCOA consists
of Dual $400$ MHz Intel Pentium II Processors in SMP configuration and
$512$ MB of memory. A single Baynetworks $24$-port fast-ethernet switch
with a backplane bandwidth of $2.5$ Gbps was used for the networking.
The whole system cost approximately \$$100,000$ (1998 US dollars).
COCOA-2~\cite{cocoa2} (Fig.~\ref{fig:cocoa2}), a successor to COCOA was also
recently built at the cost of \$$60,000$, and is a $40$-processor
rackmounted cluster (dual PIII-800s) with 20 GB memory connected via
two fast-ethernet adaptors (using Linux channel bonding). Each of the
clusters have 100 GB of usable disk space. COCOA and COCOA-2 were built
to enable the study of complex fluid dynamics problems without depending
on expensive external supercomputing resources. To get even 50,000 hrs
of CPU time in a supercomputing center is difficult, while COCOA can
offer more that 400,000 CPU hours annually, and the commodity processors
are often similar in performance to special purpose processors used
in the supercomputers.

\section{Message Passing Interface}

The Message Passing Interface (MPI)~\cite{mpi} is a standard
industry/academic API for message passing libraries, designed to be a
standard for message passing in parallel computing. Message passing is a
paradigm used widely on certain classes of parallel machines, especially
those with distributed memory.

The message passing paradigm distributes all the data among several
processes with access to local data only. To access non-local
(remote) data, a message must be explicitly sent and received, i.e.,
both the sending and receiving process must participate in the
communication. This paradigm has been extremely successful for two main
reasons~\cite{mpibook2}:

\begin{enumerate}

\item {\em Generality}: The message passing paradigm is applicable to all
kinds of parallel architectures from shared-memory supercomputers to
distributed-memory computers with high-speed communication subsystems to
heterogeneous workstation clusters on slow networks.

\item {\em Performance}: Because non-local data must be accessed by the
non-trivial and expensive way of setting up and receiving messages, the
programmer is painfully aware of the cost of using such data and is
compelled to consider better ways to distribute their data. Also, this
paradigm gives the programmer maximum control over how and when their
data is sent, which is lacking in the other paradigms. This makes
it possible to write code that is more tolerant of slow networks or
heterogeneous processors.

\end{enumerate}

Although there are many variations, the basic concept of processes
communicating through messages is well understood. Over the last ten
years, substantial progress has been made in casting significant
applications in this paradigm and each vendor has implemented its
own variant. In 1994, an MPI standard was adopted which attempts to
establish a practical, portable, efficient, and flexible standard
for message passing~\cite{mpi}. The main advantages of having a
message-passing standard are portability and ease-of-use. In a
distributed memory communication environment in which the higher level
routines and/or abstractions are build upon lower level message passing
routines, the benefits of standardization are particularly apparent.
Furthermore, the definition of a message passing standard provides
vendors with a clearly defined base set of routines that they can
implement efficiently, or in some cases provide hardware support for,
thereby enhancing scalability.

There are several open-source MPI libraries available that run
on Beowulf clusters, the most prominent being MPI CHameleon
(MPICH)~\cite{mpich} being developed by Argonne National Lab, and the
Local Area Multicomputer MPI (LAM-MPI)~\cite{lammpi} being developed
by the University of Notre Dame. Most MPI libraries are written in
C, and have C++ and FORTRAN 77/90 bindings. One of the major flaws
of most existing MPI implementations (including MPICH and
LAM-MPI) is that they are not thread-safe. Thus, it is not possible to
simultaneously perform more than one set of communications by using
MPI in a multi-threaded application. In a program using MPI, all
parallelism is explicit and the programmer is responsible for correctly
identifying parallelism and implementing the resulting algorithm using
MPI functions (of the approximately 155 that are available). The
number of tasks dedicated to run a parallel program is static and new
tasks can not be dynamically spawned during run time. Pacheco's book
on MPI~\cite{mpibook} is a good reference to start with for parallel
programming using MPI.

Figure~\ref{fig:mpi} shows a sample MPI program written in C++. As is
typical of any MPI program, it uses {\tt if} statements based on
the process rank assigned at runtime to distribute the work between
different processes. Figure~\ref{fig:mpioutput} shows the output when
this program is run on $8$ processors using the command {\tt mpirun -np
8 ./hellompi}. The program sends a character string ``Hello World'' from
the master processor (with rank 0) to all the slave processors (ranks
1 to 7) waiting to receive it. On receipt, all the processors print
the string along with their process ids. Figure~\ref{fig:mpi2} shows a
sample MPI code written in C++ to communicate a custom C++ structure
containing dynamically allocated members. Here, each dynamically
allocated member has to be sent and received separately as sending the
entire structure by using its pointer will only send the pointers to the
allocated data and not the data itself.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 
\begin{figure}[!h] 
\begin{small}
\begin{verbatim}
#include <string.h>
#include <iostream>
#include <mpi++.h>            // For MPI:: functions

int main(int argc, char *argv[])
{
  char msg[100];

  MPI::Init(argc, argv);      // Initialize MPI
  int numCPUs = MPI::COMM_WORLD.Get_size();
  int myRank = MPI::COMM_WORLD.Get_rank();

  if (myRank == 0)            // Master processor
    {
      strcpy(msg, "Hello World");
      // Send message to each processor
      for (int i = 1; i < numCPUs; i++)
      {
        int target = i;
        int tag = i;
        MPI::COMM_WORLD.Send(msg, 100, MPI::CHAR, target, 2*tag);
      }
    }
  else                        // Slave processors
    {
      // Receive message from master processor
      int source = 0;            // Master
      int tag = myRank;
      MPI::COMM_WORLD.Recv(msg, 100, MPI::CHAR, source, 2*tag);
    }

  cout << "Proc " << myRank << " says \"" << msg << "\"" << endl;

  MPI::COMM_WORLD.Barrier();      // Synchronize communication

  if (myRank == 0)
    cout << "Bye." << endl;

  MPI::Finalize();            // Finalize MPI before exiting
  return 0;
}
\end{verbatim}
\end{small}
\caption{A simple ``Hello World'' MPI program written in C++}
\label{fig:mpi}
\end{figure}

\begin{figure}[!h] 
\begin{small}
\begin{verbatim}
Proc 0 says "Hello World"
Proc 1 says "Hello World"
Proc 2 says "Hello World"
Proc 3 says "Hello World"
Proc 4 says "Hello World"
Proc 5 says "Hello World"
Proc 6 says "Hello World"
Proc 7 says "Hello World"
Bye.
\end{verbatim}
\end{small}
\caption{Output produced by MPI program in Fig.~\ref{fig:mpi} when run on $8$ processors}
\label{fig:mpioutput}
\end{figure}

\begin{figure}[!h] 
\begin{small}
\begin{verbatim}
#include "alloc.h"            // For ALLOC* and FREE* calls
#include <mpi++.h>            // For MPI:: functions

class Sample
{
public:
  Sample(int size) { Alloc(size); }
  ~Sample() { Destroy(); }
  void Alloc(int size) {
    n = size;
    ALLOC1D(&array1, n); ALLOC1D(&array2, n); // Allocate memory for "n" elements
  }
  void Destroy() {
    if (n == 0) return;
    FREE1D(&array1, n); FREE1D(&array2, n); // Free allocated memory
    n = 0;
  }
  int     n;
  char   *array1;  // Pointer to char (for dynamic memory allocation)
  double *array2;  // Pointer to double (for dynamic memory allocation)
};

void SendSample(Sample *S, int source, int target) {
  int myRank = MPI::COMM_WORLD.Get_rank();
  if (myRank == source) {    // Source (sending) processor
      // Send the size defining the structure to the target
     MPI::COMM_WORLD.Send(&S->n, 1, MPI::INTEGER, target, 101);
      // Send each dynamically allocated member separately
     MPI::COMM_WORLD.Send(S->array1, S->n, MPI::CHAR, target, 102);
     MPI::COMM_WORLD.Send(S->array2, S->n, MPI::DOUBLE, target, 103);
    }
  else if (myRank == target) { // Target (receiving) processor
      S->Destroy();  // Clear the structure
      // Receive the size defining the structure from the source
      MPI::COMM_WORLD.Recv(&S->n, 1, MPI::INTEGER, source, 101);
      S->Alloc(S->n); // Initialize the structure based on the received size
      // Receive each dynamically allocated member separately
      MPI::COMM_WORLD.Recv(S->array1, S->n, MPI::CHAR, source, 102);
      MPI::COMM_WORLD.Recv(S->array2, S->n, MPI::DOUBLE, source, 103);
    }
}
\end{verbatim}
\end{small}
\caption{MPI code written in C++ depicting communication of custom structure containing dynamically allocated members}
\label{fig:mpi2}
\end{figure}

While there are several other parallel programming paradigms such as PVM
and OpenMP available, they are not currently widely used and are not
considered here.
