##################################################################
# Introduction
##################################################################

This is a distribution of ScalaTrace 2, the next generation of the 
ScalaTrace tool that collects communication and I/O traces for parallel 
applications using the Message Passing Interface. 



##################################################################
# To compile
##################################################################

This ScalaTrace 2 package contains two components: ScalaTrace (the
tracing tool) and ScalaReplay (a replay engine for the ScalaTrace
traces). 

To compile ScalaTrace,

* cd ./record/config
* Edit Makefile.config
* cd ..
* make
* cd ..
* cd ./stackwalk/ver0
#on BG/Q machine, use unwind stackwalk###########
* cd libunwind-1.1
* ./configure
* make
* cd ..
################################################
* make 

Configurable options:

Edit ./record/libsrc/Makefile.libsrc

* RETAIN_TAGS:
n: ignore TAG for MPI_Send, MPI_Recv, etc, for better compression
y: preserve TAG

* PARAM_HISTO:
n: fully lossless tracing
y: record SRC/DEST/COUNT parameters with histograms when lossless 
   compression fails
   More configuration:
   HISTO_THRESHOLD: in ./record/common/inc/Param.h, the maximum
                    vector length for an elastic data element to 
                    remain lossless, e.g., HISTO_THRESHOLD = 30

* SIG_DIFF: 
integer: stack signature difference. This is the maximum number
of different frames allowed in two signatures

* LOOP_LCS:
n: disable approximate loop matching
y: enable approximate loop matching
   More configuration:
   MAX_LENGTH_DIFF: in ./record/common/inc/Trace.h, the maximum
                    percentage difference on length for two loop
                    iterations to be considered approximately 
                    matching, e.g., MAX_LENGTH_DIFF = 0.2 (20%)
* SKIP_IO:
n: enable IO events' tracing
y: skip(disable) IO events' tracing

* IO_TIMER:
n: disable IO_TIMER record
y: enable IO_TIMER record

* STATIC:
n: if link with dynamic mpi library, set it to be n. scalatrace will
   trace MPI IO event in MPI level.(If use ScalaReplay to replay the
   trace file, set it to be n). 
y: if link with static mpi library, set it to be y. scalatrace will
   trace MPI IO event in lower posix level.(If use darshan replay tool
   to replay, set it to be y).
Note: when it's set as y, the generated trace file can not be replayed
	by scalatrace replay, it only can be replayed by darshan replay.
	while when set as n, it only can be replayed by scalatrace replay.

To compile ScalaReplay

* cd ./replay
* Edit Makefile
* make 
* make install

Configurable options:

Edit ./replay/Makefile

* HISTO_REPLAY
n: deterministic replay, only applicable to fully lossless traces
y: probabilistic replay, applicable to both lossless and lossy 
   traces

* RAND_DEST
integer: maximum number of random destinations for each MPI task,
         e.g., RAND_DEST = 10



##################################################################
# To run
##################################################################

To use ScalaTrace, link ScalaTrace with your application:  

* ST=/path/to/ScalaTrace
* mpicxx -g -L$(ST)/record/lib -L$(ST)/stackwalk/ver0 app.c -o app 
  -lglob -lstackwalk -Wl,-wrap=open -Wl,-wrap=write -Wl,-wrap=read 
  -Wl,-wrap=open64 -Wl,-wrap=creat -Wl,-wrap=close -Wl,-wrap=lseek 
  -Wl,-wrap=lseek64 -Wl,-wrap=remove (if link with static mpi library,
  add -static at the end)

* mpirun -np <n> ./app
  the trace file will be generated at application completion, in
  the folder trace_<n>
* Compression levels:
  libdump.a: trace with no compression
  libnode.a: trace with intra-node loop compression
  libglob.a: trace with both intra-node and inter-node compression

To use ScalaReplay, use an application trace (T) as the input

* SR=/path/to/ScalaTrace/replay
* mpirun -np <n> $(ST)/replay /path/to/trace/T



##################################################################
# Example
##################################################################

See ./test/1dstencil (without IO)
See ./test/iosample (with IO)




