%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 
\chapter{Computational Steering}\label{chapter:steering}

%\section{Introduction}

While running a complex parallel program on a HPC system, one often
experiences several major difficulties in observing output or
information about the ongoing state of the simulation. Usually, the way
the simulation is implemented severely limits the interaction with the
program during the execution and makes the visualization and monitoring
slow and cumbersome (if at all possible). This is especially the case
if the visualization needs to be carried out on a different system
(such as the RAVE). In particular, some of the questions pertaining to
visualization that arise in the simulation of the wake-vortices are the
following.

\begin{itemize}
\item How do we get the current location of the wake-vortices shed by
each aircraft for visualization on the RAVE while the parallel simulation
is running on a remote Beowulf Cluster?

\item How do we keep the simulation on the Beowulf Cluster updated about
the constantly changing weather conditions and the trajectories of the
aircraft?

\end{itemize}

These questions are of critical importance to this research as the
predictions by the wake-vortex code need to be known in real-time for
the ATC system to take appropriate action. 
The activity of periodically obtaining
the location of the wake-vortices is referred to as ``monitoring,''
which is defined as the observation of a program's behavior at specified
intervals of time during its execution. On the other hand, the weather
condition at the airport is continuously changing and both the number
and the trajectories of aircraft change as they take-off and land.
Thus, there is a need to update the simulation based on the state
factors. This process of updating ongoing simulation is referred to as
``steering''. The above questions can thus be rephrased as ``How do we
monitor and steer our wake-vortex simulation?''.

Software tools which support these activities are called (computational)
steering environments. These environments typically operate in three
phases: instrumentation, monitoring, and steering~\cite{falcon}. 
\begin{itemize}

\item Instrumentation is the phase where the application code is
modified to add monitoring functionality. This phase involves
``registration'' of relevant application data so that it can be accessed by
the remote monitoring and steering client.

\item Monitoring phase requires the program to run with some
initial input data, the output of which is observed by retrieving
important data about the program's state change. Analysis of this data
gives more knowledge about the program's activity. 

\item During the steering phase, the user modifies the program's
behavior (by modifying the input) based on the knowledge gained during
the previous phase by applying steering commands, which are injected
on-line, so that the application need not be stopped and restarted.

\end{itemize}

\section{Portable Object-oriented Scientific Steering Environment (POSSE)}

The steering software developed here, {\em Portable Object-oriented
Scientific Steering Environment} (POSSE) \cite{posse}, is a
general-purpose, lightweight, portable software system based on a simple
client\footnote{A program that sends queries over the network and accepts corresponding responses}/server\footnote{A program that provides a network service by answering related queries} model. It uses an approach similar to Falcon~\cite{falcon}
and ALICE Memory Snooper~\cite{alice_mem_snooper}. Falcon was one of
the first systems to use the idea of threads and shared memory to
serve registered data efficiently. ALICE is an API designed to help in
writing computational steering, monitoring and debugging tools developed
at Argonne National Lab. POSSE consists of a steering server on the
machine running the simulation that controls registered application
data, and coupled to any number of steering clients that provides the
user interface and control facilities to interact with the steering
server to access the data. The steering server is created as a separate
thread of the application to which local monitors forward only those
registered data that are of interest to steering activities. A steering
client receives run-time information from the application, conveys this
information to the user, accepts steering commands from the user, and
transmits these commands back to the application. The communication
between a steering client and a server are implemented using UNIX
sockets and the threading is implemented using POSIX\footnote{Portable
Operating System Interface standard} threads. POSSE has been written
completely in C++, using several of C++'s advanced object-oriented
features making it fast and powerful, while hiding most of the code
complexities from the user.

Figure~\ref{fig:posse_class} shows the inheritence diagram for the POSSE
class structure and Fig.~\ref{fig:uml} shows a UML\footnote{Unified
Modeling Language: used for modeling object-oriented applications}
diagram depicting the software design of the POSSE code. {\tt TCPSocket}
is the base class that handles all the low-level communication
consisting of contiguous byte segments using UNIX sockets. {\tt
RemoteSocket} is an inherited class that extends the {\tt TCPSocket}
functionality by providing functions for higher-level communication of
variables, arrays and structures. The inherited classes {\tt DataClient}
and {\tt DataServer} then add client and server functionalities,
respectively. {\tt DataClient} provides client-side functions for
accessing registered data, while {\tt DataServer} adds server-side
functionality for data registration and coherence. {\tt DataServerMPI}
inherits from {\tt DataServer} and provides enhancements for parallel
codes that use MPI.

\myfigureW{figs/posse_class.ps}{posse_class}{15cm}
{Inheritance diagram for POSSE classes}
\myfigureW{figs/uml.ps}{uml}{16cm}
{UML Diagram for POSSE classes}

Figure~\ref{fig:steering} shows
a schematic view of how POSSE may be used. An on-going scientific
simulation is running on a remote Beowulf computing cluster. Any number
of remote clients can query/steer registered data from the simulation
from the DataServer thread. Two clients are depicted: 
(1) a visualization client, and 
(2) a GUI client that provides a simple user interface to all
registered simulation data. 
The visualization application can be used to interactively monitor a
dataset at various time intervals. The GUI application can be used to
steer the simulation by changing registered parameters.

\myfigureW{figs/steering.ps}{steering}{12cm}
{A schematic view of POSSE}

POSSE is designed to be extremely lightweight (has very low
computational overhead), portable (runs on all Win32 and POSIX-compliant
UNIX platforms including Linux) and efficient. It deals with
byte-ordering and byte-alignment problems internally and also provides
an easy way to handle user-defined classes and data structures. It is
also multi-threaded, supporting several clients simultaneously. POSSE
supports parallel simulations that use the Message Passing Interface
(MPI)~\cite{mpi} library. The biggest enhancement of POSSE over existing
steering systems is that it provides the same features of these systems,
yet is extremely easy to use, making augmentation of any existing C/C++
simulation code possible in a matter of hours. POSSE makes extensive use
of C++ classes, templates and polymorphism to keep the user Application
Programming Interface (API) elegant and simple to use.

Figures~\ref{fig:steeringserver} and~\ref{fig:steeringclient}
illustrate a simple, yet complete, POSSE client/server program in
C++. As seen in Fig.~\ref{fig:steeringserver}, registered data on the
steering server (which are marked {\em read-write}) are protected using
binary semaphores when they are being updated in the computational code.
Also, due to the automatic compile-time type-inferencing capability
of C++ templates, the type of the data being registered need not
be specified, thus making it easier and less error-prone for the
programmer. User-defined data structures are handled by a simple
user-supplied pack and unpack subroutine that call POSSE data-packing
functions to tackle the byte-ordering and byte-alignment issues. 

The programmer need not know anything about the internals of threads,
sockets or networking in order to use POSSE effectively. POSSE also
allows for a simulation running on any parallel or serial computer to be
monitored and steered remotely from any machine on the network using a
cross-platform Graphical User Interface (GUI) utility.
%Among other applications, POSSE has been successfully used to enhance
%an existing parallel Computational Fluid Dynamics (CFD) code to perform
%visualizations of large-scale flow simulations~\cite{pumasteering}
%(Chapter~\ref{chapter:cfd}).

\begin{figure}[!p] 
\begin{small}
\begin{verbatim}
#include "dataserver.h"

const int MAX_ITER = 2000;
int dummyInt = 0, n1, n2;
double **dyn2D;

REGISTER_DATA_BLOCK()   // Register global data
{
  REGISTER_VARIABLE("testvar", "rw", dummyInt);
  REGISTER_DYNAMIC_2D_ARRAY("dyn2D", "ro", dyn2D, n1, n2);
}

int main(int argc, char *argv[])
{
  DataServer *server = new DataServer;
  
  if (server->Start(4096) != POSSE_SUCCESS) // Start Server thread
      {
       delete server;
       exit(-1); 
      }
  n1 = 30; n2 = 40;
  ALLOC2D(&dyn2D, n1, n2);

  for (int iter = 0; iter < MAX_ITER; iter++) {
         server->Wait("dyn2D"); // Lock DataServer access for dyn2D
           /*
             Compute(dyn2D);    // Update dyn2D with new values
           */
         server->Post("dyn2D"); // Unlock DataServer access for dyn2D
   }
  FREE2D(&dyn2D, n1, n2);
  delete server;
}
\end{verbatim}
\end{small}
\caption{A simple, complete POSSE server application written in C++}
\label{fig:steeringserver}
\end{figure}
\begin{figure}[!p]
\begin{small}
\begin{verbatim}
#include "dataclient.h"

int main(int argc, char *argv[])
{
  DataClient *client = new DataClient;
  double **dyn2D;

  // Connect to DataServer
  if (client->Connect("cocoa.ihpca.psu.edu", 4096) != POSSE_SUCCESS)
       {
        delete client;
        exit(-1);
       }
  client->SendVariable("testvar", 100); // Send new value for "testvar"
  int n1 = client->getArrayDim("dyn2D", 1);
  int n2 = client->getArrayDim("dyn2D", 2);
  ALLOC2D(&dyn2D, n1, n2);
  client->RecvArray2D("dyn2D", dyn2D);
  /*
    Use(dyn2D);  // Utilize dyn2D
  */
  FREE2D(&dyn2D, n1, n2);
  delete client;
}
\end{verbatim}
\end{small}
\caption{A simple, complete POSSE client application written in C++}
\label{fig:steeringclient}
\end{figure}

\section{Real-Time Implementation Issues}
             
Real-time systems consist of several networked machines, often with
differing hardware and software communicating with each other via
messages.
 For message communication to work correctly, the message
formats have be defined unambiguously. In many systems this task is achieved
simply by using arrays and/or structures to implement the message
format. While this may be a simple approach, it has its own pitfalls.
Some of the portability problems plaguing this are:
\begin{enumerate}
\item byte-ordering or endianness, and
\item byte-alignment issues.
\end{enumerate}
Another problem that arises when periodically exchanging data that
is continually evolving at either end is that of ``data coherence.''
Also, if the system is to be run on a parallel machine, we have to
worry about ``computational overhead and scalability.'' The sections
below describe each of these issues in detail and discusses ways to
resolve them.

\subsection{Endianness}
When transporting data over a computer network from one machine to
another, one has to ensure that the represention of this data is uniform
across both the machines. However, this is not always the case due to
differing endianness between the machines. Endianness is the attribute
of a hardware system that indicates the order of the bytes representing
basic data types (e.g., integers, floats and their variants). Endianness
are of two types: {\em big-endian} and {\em little-endian}. A big-endian
system represents the data types in a left to right order with the most
significant byte (MSB) present on the left. Conversely, a little-endian
system represents the data types in a left to right order with the least
significant byte (LSB) present on the left. Examples of big-endian
machines are Cray, HP, IBM, Mac, SGI and Sun, and those of little-endian
machines are PC (x86), Alpha and VAX.

Consider the hexadecimal representation of the 4-byte (32-bit) integer $2030005539$:
$$
2030005539 \rightarrow \textrm{78FF6D23}
$$
In the big-endian representation, the byte-ordering is:
$$
\textrm{Big-endian:} \hspace{0.4cm} \textrm{78FF6D23} \rightarrow \overbrace{\textrm{78}}^{\textrm{MSB}}\hspace{0.4cm} \textrm{FF}\hspace{0.4cm} \textrm{6D}\hspace{0.4cm} \overbrace{\textrm{23}}^{\textrm{LSB}} 
$$
In the little-endian representation, the byte-ordering is reversed and is:
$$
\textrm{Little-endian:} \hspace{0.4cm} \textrm{78FF6D23} \rightarrow \overbrace{\textrm{23}}^{\textrm{LSB}}\hspace{0.4cm} \textrm{6D}\hspace{0.4cm} \textrm{FF}\hspace{0.4cm} \overbrace{\textrm{78}}^{\textrm{MSB}} 
$$
Hence, a memory dump for the code-segment below:
\begin{verbatim}
char  a = 1;
char  b = 2;
short c = 255;   // 0x00FF
int   d = 23245; // 0x01EBD405
\end{verbatim}
will look like the following under the different endian systems:
$$
\begin{array}{lcccccccc}
\textrm{Little-Endian:} & \textrm{01} & \textrm{02} & \textrm{FF} & \textrm{00} & \textrm{05} & \textrm{D4} & \textrm{EB} & \textrm{01} \\
\textrm{Big-Endian:} & \textrm{01} & \textrm{02} & \textrm{00} & \textrm{FF} & \textrm{01} & \textrm{EB} & \textrm{D4} & \textrm{05} \\
\end{array}
$$
Figure~\ref{fig:endianness} illustrates a simple C++ function to
determine the endianness of any given system. The function works by
initializing a 2-byte data field ({\tt short}) with a known hex value
(0x4142) with differing values for each of the byte locations. A
character pointer is then attached to the beginning of its memory
location. If this pointer, which points to the first and left-most byte
for the 2-byte data contains the hex value 0x41 then the system is
determined as big-endian, and if it contains the hex value 0x42 then it
is determined as little-endian. 
Figure~\ref{fig:swap} illustrates
another general C++ function to reverse the byte-ordering of any data
type by swapping the bytes at opposite ends. This can be used to change
the representation of any data type from little-endian to big-endian and
vice versa.

\begin{figure}[!p]
\begin{small}
\begin{verbatim}
bool isBigEndian()
{
  short s = 0x4142;      // 2-byte data-type
  char *leftmostbyte = (char *) &s;

  if (leftmostbyte[0] == 0x41 && leftmostbyte[1] == 0x42)
      return true;      // Big-endian
  if (leftmostbyte[0] == 0x42 && leftmostbyte[1] == 0x41)
      return false;     // Little-endian
// Should never reach here!
}
\end{verbatim}
\end{small}
\caption{A simple C++ function to determine the endianness of any system}
\label{fig:endianness}
\end{figure}

\begin{figure}[!p]
\begin{small}
\begin{verbatim}
template <class Etype>
Etype swapbytes(Etype in)
{
  Etype out;
  char *p_in = (char *) &in;
  char *p_out = (char *) &out;
  static int size = sizeof(Etype);

  for (int i = 0; i < size; i++)
       p_out[i] = p_in[size-1-i];

  return out;
}
\end{verbatim}
\end{small}
\caption{A simple C++ function to reverse byte-ordering of any data type}
\label{fig:swap}
\end{figure}

As can be concluded from the above explanation, the endianness issue is only
associated with binary data comprising of any multibyte data type. Since
a {\tt char} or an {\tt unsigned char} are represented using 1 byte on
all known systems, they are independent of byte-ordering. Hence, ASCII data
comprising of characters are considered universally portable.

Endianness issues have to be resolved efficiently and unambiguously when
dealing with machines with differing byte-order. For this, the computer
network defines its own byte-order, namely the {\em network byte order},
which is big-endian. All data transported over any computer network is
converted from the host byte-order to the network byte-order. On the
receiving end, the data is converted from the network byte-order to
the target host's byte-order. This can lead to conversion overhead for
little-endian machines which can reduce the effective network bandwidth
by as much as $10-15\%$. If majority of the machines participating
in the communication are big-endian, then this is not of concern.
However, if majority of the machines participating in the communication
are little-endian, then the overhead can become significant as there
will be a conversion required from big-endian to little-endian (or
vice-versa) for every machine on the network that is little-endian. To
avoid this overhead, POSSE can use either little-endian or big-endian as
the network byte-order, thus reducing the overall conversion overhead
if knowledge about the systems communicating are known a priori. For
example, using little-endian as the network byte-order among a network
of Intel PCs (little-endian) reduces overhead greatly as no byte
re-ordering of data needs to be performed.

\subsection{Byte-alignment}
The byte-alignment problem crops up due to the fact that different
processors/compilers might define the same structure differently, thus
causing incompatibility in the interface definition. Here, alignment
is defined as the placing of data and code in the computer memory in
addresses that are more efficient for the hardware to access (at the
price of wasting some memory).

32-bit microprocessors typically organize memory in 32-bit (4-byte)
increments.
Memory is accessed
by performing 32-bit bus cycles. Since memory is organized in 4-byte
increments, these 32-bit bus cycles can only be performed at addresses
that are divisible by 4. Table~\ref{table:alignment} shows a snapshot of
memory with two 4-byte data $A$ and $B$, where $A$ comprising of bytes
$A_1$, $A_2$, $A_3$ and $A_4$ is aligned with the 4-byte boundary, and
$B$ comprising of bytes $B_1$, $B_2$, $B_3$ and $B_4$ is not.

\begin{table}[h!]
\centerline{
\begin{tabular}{|c|c|c|c|c|} \hline
&{\bf Byte 1 }&{\bf Byte 2}&{\bf Byte 3}&{\bf Byte 4} \\ \hline
0x00000000 & & & & \\ \hline
0x00000004 & $A_1$ & $A_2$ & $A_3$ & $A_4$ \\ \hline
0x00000008 & & & & \\ \hline
0x0000000B & & $B_1$ & $B_2$ & $B_3$ \\ \hline
0x00000010 & $B_4$ &  &  & \\ \hline
\end{tabular}}
\caption{Byte-alignment problem}
\label{table:alignment}
\end{table}

The reasons for not permitting misaligned data reads and writes are
clearly seen from the table. For example, the aligned data $A$ can
be read by the processor in a single bus cycle starting at address
0x00000004. If the same processor now attempts to access the data $B$
at address 0x0000000C, it cannot do it in a single bus cycle. The
processor will have to issue two different reads at address 0x0000000B
and 0x00000010 to read the complete 4-byte data. Thus it takes twice the
time to read a misaligned 32-bit data. Often times, misaligned addresses
may generate bus errors and abort the program. Older computers (e.g.,
VAX systems), can handle misaligned memory accesses more gracefully, but by
degrading the performance. This approach was mainly due to the fact that memory
was once very costly, and computer designers could not afford to waste
even a small amount on alignment.

Code alignment is usually controlled by the compiler.  Let us consider
the following user-defined data-structure in C:
\begin{small}
\setstretch{1.2}
\begin{verbatim}
struct myStruct
{
  short v1;   // 2-byte
  char  v2;   // 1-byte
  int   v3;   // 4-byte
  char  v4;   // 1-byte
  short v5;   // 2-byte
};
\end{verbatim}
\setstretch{1.8}
\end{small}
For a 32-bit processor, the compiler will likely use the following
padded definition for the data-structure:
\begin{small}
\setstretch{1.2}
\begin{verbatim}
struct myStruct
{
  short     v1;  // 2-byte
  char      v2;  // 1-byte
  char pad1[1];  // Pad to start the following "int" 4-byte boundary
  int       v3;  // 4-byte
  char      v4;  // 1-byte
  char pad2[3];  // Pad to start the following "short" at a 4-byte boundary
  short     v5;  // 2-byte
  char pad3[2];  // Pad to complete aligning at a 4-byte boundary
};
\end{verbatim}
\setstretch{1.8}
\end{small}
In the above example, the compiler has added pad bytes to enforce byte
alignment rules of the target processor. If the above data-structure
was used in a different compiler/microprocessor combination, the pads
inserted by that compiler might be different. Thus, two applications
using the same data-structure definition may have different sizes and
may be incompatible with each other.

POSSE tackles the byte-alignment issue (and the byte-ordering issue)
by having the user provide a custom packing and unpacking function for
every user-defined data-structure that needs to be communicated over the
network. These function individually pack and unpack each field in the
structure using macros provided by POSSE. Figure~\ref{fig:packunpack}
illustrates this with a simple user-defined data-structure {\tt
Trajectory} along with the functions {\tt packStruct()} and {\tt
unpackStruct()} to pack and unpack the structure during the
communication. The POSSE macros also convert the host byte-ordering to
the network byte-ordering before packing the data, and back from the
network byte-ordering to the host byte-ordering after unpacking the
data. Thus the resulting contiguous data-segment ({\tt unsigned char
*data}) containing the data-structure is unambiguous and portable. An
advantage of this approach is that the entire structure need not be
packed; only parts of the structure of interest to the client can be
packed and unpacked, thus reducing time to convert and transport the
data.

\begin{figure}[!p]
\begin{small}
\begin{verbatim}
typedef struct { 
   int i; 
   double d; 
   float a[200][400]; 
} Trajectory;

int packStruct(Trajectory *T, unsigned char **dataptr, int &totsize)
{
  totsize = sizeof(T->i) + sizeof(T->d) + sizeof(T->a);
  unsigned char *data = new unsigned char[totsize]; 
  *dataptr = data;
  int ptr = 0;

  PACK_VARIABLE(T->i, data, &ptr);
  PACK_VARIABLE(T->d, data, &ptr);
  PACK_2D_ARRAY(T->a, data, &ptr);

  return ptr;
}

int unpackStruct(Trajectory *T, unsigned char *data, int size)
{
  int ptr = 0;
 
  UNPACK_VARIABLE(&(T->i), data, &ptr);
  UNPACK_VARIABLE(&(T->d), data, &ptr);
  UNPACK_2D_ARRAY(T->a, data, &ptr);

  return ptr;
}
\end{verbatim}
\end{small}
\caption{Packing and unpacking of a custom C++ structure}
\label{fig:packunpack}
\end{figure}

\subsection{Data coherence}
Whenever there is an exchange of constantly changing data between
networked machines, one has to worry about the problem of maintaining
data coherence. There are several situations that can lead to
data inconsistencies.

\begin{enumerate}
\item A request for data arrives while it is being modified. If the
request is answered immediately, the data sent may be incoherent,
especially if the data is an array or a structure which may require
several milliseconds to update. A part of the data sent will be in the
new state after modification while the rest will be in the old
state before modification.

\item Multiple clients may want to modify the same portion of a
{\em read-write} data on a server simultaneously. If this is not
controlled by some means, parts of data modified by one client can
be overwritten by the other client leading to inconsistent
data on the server-side. If another client requests this data
immediately thereafter (or worse, during the write operation), it
will receive incoherent data.

\item A server is utilizing a particular {\em read-write} data for a
computation in a loop. It expects this data to remain unchanged during
any given computation loop (e.g., time step iteration) to yield correct results.
If during any particular iteration, a client tries to modify the data
and succeeds, then the rest of the computation for that iteration will
use the modified data and may produce inconsistent results.

\end{enumerate}

These data coherence problems are dealt with in POSSE through the use of binary
semaphores. A semaphore is a classic method for restricting access
to shared resources in a multi-processing or multi-threading
environment~\cite{osbook}. Since each client request in POSSE is handled
by a separate thread on the server-side (see Fig.~\ref{fig:steering}),
chances of conflict between threads is likely to occur.
On the server-side, binary semaphores are used to lock and unlock
{\em read-write} data (also known as {\em critical data}) when they
are being modified. This has to be done by the user using the lock
associated with that data handle. This is illustrated 
in  Fig.~\ref{fig:steeringserver}
with the {\tt
Wait("dyn2D")} and {\tt Post("dyn2D")} commands used in the C++ server
code.
 At any given time, at most one
thread can acquire the lock by issuing the {\tt Wait()} command. While
the lock is held by the user code, no DataServer thread can acquire the
lock which is required in order to allow modification to the critical
data. Once the user releases the lock by issuing the {\tt Post()}
command, any DataServer thread may acquire the lock and proceed to
modify the data. Modification for server-side data is first received
into a buffer and then copied to the target location after a lock is
acquired for the target data. This is done to minimize the holding
period of the lock by excluding the communication time required to
receive the data over the network which may be significant. Long holding
periods for a lock can lead to poor response times for competing threads
waiting for the same lock.

%Discusses pros and cons of approaches used. Highly templatized code
%can lead to bloated binaries slowing things down if used often....can
%also take huge amounts of time to compile, etc. Other issues.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 
%\section{Scalability and Dimensional Reduction}
\section{Overhead and Scalability}

One of the most important issues with any steering software that is
used in conjunction with parallel programs is that of scalability. It
is essential that the processor overhead due to the steering thread
be small relative to the primary computational thread. When this is
not the case, the primary computational thread will compete with the
steering thread for CPU cycles slowing down the entire simulation. Also,
it is expected that the computational overhead be uniform across all
the compute nodes. If not, due to the nature of SPMD parallelism, the
node having the highest overhead will dominate slowing down all the
other nodes along with it. 

In POSSE, the blocking socket {\tt accept()} call is used to wait for
a client connection and a blocking socket {\tt recv()} call is used to
wait for requests from an established client connection. Since these
calls are blocking, they work directly with the network (TCP/IP) layer
and hence do not use polling\footnote{Periodic checking in an infinite loop} (which can be very inefficient and can lead
to high processor overhead). Thus, when no {\tt DataClient} requests
are being processed by the {\tt DataServer}, the overhead is limited to
that of the blocking {\tt accept()} and/or {\tt recv()} calls, which
is extremely low ($< 0.1$\% of total processor time). When there are
requests being handled, the overhead varies with the number of requests
and the type of requests ({\em read} vs. {\em write}). This overhead is
primarily due to the spawning of new server threads to answer requests
from each client, and due to the constant locking and unlocking of
critical data (if any). A detailed performance analysis of this overhead
is presented in Section~\ref{sec:results}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 
\section{Experimental Results}	\label{sec:results}

POSSE has been extensively tested using various platforms for stability
and performance. In this section, tests demonstrating both the single
and multiple client performance of POSSE are presented.

\subsection{Single Client Performance}
Fig.~\ref{fig:bandwidth2} shows a plot of the effective network
bandwidth achieved by varying the size of a dynamic 1-D array requested
by a steering client. These tests were carried between machines
connected via a Fast Ethernet connection having a peak-theoretical
network bandwidth of 100 Mbps. The communication time used to calculate
the effective bandwidth includes the overheads for byte-ordering, data
packing and other delays introduced by the querying of the registered
data. The average latency for query of any registered data by the client
has been found to be $38$ ms. As can be seen, there is a noticeable
decrease in the bandwidth (about $10$ Mbps) when communicating between
machines with different byte-ordering (i.e., little endian vs. big
endian) as opposed to machines with the same byte-ordering. This
difference reflects the overhead involved in duplicating the requested
data and converting it into the byte-order of the client machine for
communication. In the same byte-order case, as the size of the requested
data increases to about $5$ MB, the effective bandwidth approaches
$80$ Mbps, which is $80 \%$ of the peak-theoretical bandwidth. The
source code for the programs used to carry out the test can be found in
Appendix~\ref{ch:scp}.


\subsection{Multiple Client Performance}
Fig.~\ref{fig:bandwidth1} shows a plot of the effective bandwidth
achieved by varying the number of clients simultaneously requesting
data. In this test, both the clients and the server were machines with
the same byte-ordering. The server had a registered 4-D array with
$200,000$ {\em double} elements ($1.6$ MB of data) but did not perform
any computation in the main thread. There was an idle loop included in
the {\tt main()} function. All the clients were then run simultaneously
from two remote machines on the same network and were programmed to
request the $1.6$ MB 4-D array from the server. The effective bandwidth
in this case is obtained by dividing the total amount of data served
by the server with the total wall-clock time required to serve all the
requests. It can be seen that the network performance of POSSE is very
good ($84$ Mbps) even when dealing with over $500$ client requests
simultaneously. The source code for the programs used to carry out the
test can be found in Appendix~\ref{ch:mcp}.

For the wake-vortex simulation system, the amount of data
communicated to the client after every update is 420 bytes for every
aircraft and 56 bytes for every vortex element. For 10 aircraft each
having $2,000$ elements tracked, this amounts to $1.12$ MB of data.
From Fig.~\ref{fig:bandwidth2}, we can see that this corresponds to
a data-rate of approximately $62$ Mbps, or $145$ ms of communication
time. Thus, we can get updated data at a rate of almost 7 fps from the
server. The wake-vortex simulation runs with a $\Delta t$ of 0.2 seconds
which can be maintained for up to $2,000$ vortex elements per aircraft
on COCOA-2. The parallel code has very good scalability for up to 15
processors (tracking 15 aircraft), after which it linearly deteriorates
due to the overhead borne by the master for distributing and collecting
data from the slave nodes. At this point, the simulation has only been
qualitatively checked and seems to be consistent with the theory. The
simplification of using $k$ neighbors for induced-velocity computation
works very well with an error of less than $1\%$ when compared to the
original $O(N^2)$ case. Since the weather conditions play a substantial
role in the determination of the vortex decay, a more sophisticated
weather model like the one used in AVOSS will definitely improve the
accuracy of the simulations.

\myfigureBW{figs/bandwidth2.ps}{figs/bandwidth.ps}{bandwidth2}{bandwidth1}{11cm}
{Effective Network Bandwidth vs. Size of data requested by client}
{Effective Network Bandwidth vs. Number of simultaneous clients}

\subsection{Symmetric Multi-Processor (SMP) vs. Uni-Processor Server}
Figure~\ref{fig:smp} shows the effect of Symmetric Multi-Processor
(SMP) vs. Uni-Processor (UP) for the machine running the server-side
application. This was obtained by running a variation of the server-side
code used for the multiple client performance test in the section above.
The server was run on an dual PIII-400 Mhz SMP machine running Linux.
For the UP tests, the Linux kernel without SMP support was used to boot
the machine resulting in the detection of only one processor. In the
server for this test, an actual computation loop was included instead
of an idle loop in the {\tt main()} function. All the clients were then
invoked simultaneously. Once all the client requests were answered by
the server, the number of computation loops performed per second were
noted. As can be seen clearly, the 2-processor SMP machine performs
approximately $48\%$ more computations per unit time as compared to the
UP machine while answering the same number of client requests. This is
due to the multi-threaded nature of the server application which scales
well on a multi-processor architecture. A single processor machine
cannot cope up with several simultaneous client requests without slowing
down the computation. The source code for the programs used to carry out
the test can be found in Appendix~\ref{ch:smp}.

\myfigureW{figs/smp.ps}{smp}{11cm}
{Effect of SMP vs UP machine on the server}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 
