====== Running Auryn simulations on the Intel® Xeon Phi™ Product Family (MIC architecture) ======

The following section is experimental. To compile Auryn in native mode for the [[http://www.intel.com/content/www/us/en/processors/xeon/xeon-phi-detail.html|Intel Xeon Phi]] cards the following steps worked. Requirements: First, use the newest Auryn development version, which does not compile against GSL any more. This saves you some extra trouble in compiling GSL. Second, you need the Intel Composer suite. Third, you need the boost libraries (see [[required libraries]]) compiled for the Intel MIC architecture ideally using the Intel MPI implementation which is native to Xeon and the Xeon Phis. 

This has been tested for the development version available on [[https://github.com/fzenke/auryn|GitHub]] as commit ''5b02d058844ea9b6344cc4e94491c10cf5ea0522''.

===== Compiling Boost for Xeon Phi/MIC =====

For the most part you can follow one of Boosts tutorials for the compilation [[http://www.boost.org/doc/libs/1_55_0/doc/html/mpi/getting_started.html#mpi.config]]. In particular I set ''user-config.jam'' to contain the line 
<code>
using mpi : /opt/intel/impi/4.1.3.048/bin/mpigxx -cxx=/opt/intel/bin/icpc -static_mpi : -L/opt/intel/impi/4.1.3.048/lib ;
</code>
to enable MPI and to tell the build toolchain Jam which compiler to use. Note that you might have to adapt the version numbers in this example to your actual version.

Building of boost was then achieved by using a command line similar to the following. It is important to pass the ''-mmic'' arguments to tell the Intel compiler that it is cross compiling for the Xeon Phi:
<code shell>
./b2 toolset=intel  --disable-icu --without-iostreams cflags="-mmic" cxxflags="-mmic -I/opt/intel/impi/4.1.3.048/mic/include/" linkflags="-mmic" --debug-configuration
</code>

===== Preparing Auryn for Xeon Phi/MIC =====

To run smoothly on the architecture and to make efficient use of vectorization you need to set the following switches in the ''auryn_definitions.h'' in the ''src/'' directory of Auryn. In particular you have to deactivate prefetching and activate SIMD instructions with Intel CILK Plus support. 
To do so find the following precompile directives in ''auryn_definitions.h'' and un/comment them accordingly:
<code cpp>
// #define CODE_ACTIVATE_PREFETCHING_INTRINSICS
#define CODE_USE_SIMD_INSTRUCTIONS_EXPLICITLY
#define CODE_ACTIVATE_CILK_INSTRUCTIONS
</code>

===== Building Auryn for Xeon Phi/MIC =====

With this done one simply has to create a new subdir in the Auryn build directory and add the following Makefile:

<code Makefile>
.SECONDARY:

CC = /opt/intel/impi/4.1.3.048/bin/mpicxx -cxx=/opt/intel/bin/icc
CXX = /opt/intel/impi/4.1.3.048/bin/mpicxx -cxx=/opt/intel/bin/icc

CFLAGS= -ansi -Wall -pipe -O3 -mmic -static_mpi -static-intel -vec-report2 -pedantic   -I$(SRCDIR) -I$(DEVDIR)
LDFLAGS= -L/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic/  -lboost_program_options -lboost_serialization -lboost_mpi -limf -lsvml -lirng -lintlc

include ../Makefile.include
</code>

Again the path variables will probably need to be updated to your needs. Furthermore depending on where you installed the Boost-for-MIC libraries you might have to add a cflag ''-I/lcncluster/zenke/lib/build/boost_1_55_0/'' and similarly a linkflag ''-L/yourpath/boost_1_55_0/stage/lib''. 

Now you are almost set for compilation. However, before you compile ''icc'' generally requires you to update your ''LD_LIBRARY_PATH'' and ''MIC_LD_LIBRARY_PATH''. This is most conveniently achieved by
<code shell>
source /opt/intel/impi/4.1.3.048/bin/mpivars.sh
source /opt/intel/composer_xe_2013_sp1.2.144/bin/compilervars_arch.sh  mic
</code>

Invoking ''make'' will now build the simulation binaries.


===== Running Auryn simulations on Xeon Phi/MIC =====

To run a simulation (e.g. sim_background) in native mode on a Xeon Phi card you need to copy the binary and the required shared libraries to the card. This essentially follows the steps outlined in this Intel tutorial [[https://software.intel.com/en-us/articles/using-the-intel-mpi-library-on-intel-xeon-phi-coprocessor-systems]].

Finally the code can be run on the coprocessor card:
<code shell>
[xyz@srv1-mic0 tmp]# mpirun -n 60 ./sim_background --dir /tmp --simtime 10
</code>


Enjoy!