HDTrace is an environment which allows to trace and simulate the behavior of MPI programs on a cluster. It explicitly includes support to trace internals of MPICH2 and the parallel file system PVFS. With this support it enables to localize inefficiencies, to conduct research on new algorithms and to evaluate future systems. Simulation provides upper bounds of expected performance and helps to assess observed performance as potential performance gains of optimizations can be approximated.
In this paper the environment is introduced and several examples depict how it assists to reveal internal behavior and spot bottlenecks. In an example with PVFS the inefficient write-out of a matrix diagonal could be either identified by inspecting the PVFS server behavior or by simulation. Additionally the simulation showed that in theory the operation should finish 20 times faster on our cluster – by applying correct MPI hints this potential could be exploited.

@techreport{HATASEOAAS11,
	author	 = {Julian Kunkel},
	title	 = {{HDTrace – A Tracing and Simulation Environment of Application and System Interaction}},
	year	 = {2011},
	month	 = {01},
	publisher	 = {Research Group: Scientific Computing, University of Hamburg},
	address	 = {Deutsches Klimarechenzentrum GmbH, Bundesstraße 45a, D-20146 Hamburg},
	series	 = {Research Papers},
	number	 = {2},
	abstract	 = {HDTrace is an environment which allows to trace and simulate the behavior of MPI programs on a cluster.
It explicitly includes support to trace internals of MPICH2 and the parallel file system PVFS.
With this support it enables to localize inefficiencies, to conduct research on new algorithms and to evaluate future systems.
Simulation provides upper bounds of expected performance and helps to assess observed performance as potential performance gains of optimizations can be approximated.

In this paper the environment is introduced and several examples depict how it assists to reveal internal behavior and spot bottlenecks.
In an example with PVFS the inefficient write-out of a matrix diagonal could be either identified by inspecting the PVFS server behavior or by simulation.
Additionally the simulation showed that in theory the operation should finish 20 times faster on our cluster -- by applying correct MPI hints this potential could be exploited.},
}