-
Notifications
You must be signed in to change notification settings - Fork 0
/
README
280 lines (222 loc) · 10.5 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
// The LANL computer code number is
// CLAMR -- LA-CC-11-094
////////////////
// GIT Commands
////////////////
To checkout the code
% git clone [email protected]:losalamos/CLAMR.git
////////////////////////
// Useful bash commands
////////////////////////
// logistics
locate [file] // find file in directory
history // see history of bash commands
source [file] // read this file (in a shell form)
// code checking
valgrind // check memory errors
// To run valgrind with libtool built code
libtool --mode=execute valgrind --tool=memcheck --leak-check=full --show-reachable=yes --track-origins=yes --leak-resolution=high --num-callers=20 --log-file=vgdump clamr_cpuonly -n 16 -i 1 -t 1
ddd // debugger
totalview (there is a module on yellowrail and turing)
// code compiling
// We have switched to cmake so the configure step is
cmake . // in-tree build
cmake <path-to-src> // out-of-tree build
cmake -DCMAKE_BUILD_TYPE=debug <path-to-src>
cmake -DCMAKE_BUILD_TYPE=release <path-to-src>
cmake -DGRAPHICS_TYPE=None <path-to-src> // Turn off real-time graphics
cmake -DGRAPHICS_TYPE=OpenGL <path-to-src> // Default
cmake -DGRAPHICS_TYPE=MPE <path-to-src> // Alternate real-time graphics
cmake -DPRECISION_TYPE=full_precision // full double precision
cmake -DPRECISION_TYPE=mixed_precision // intermediates double, arrays single
cmake -DPRECISION_TYPE=minimum_precision // all single precision
// To build:
make
// or for a specific executable
make clamr_cpuonly
// To run the unit tests
make check
// or for a specific package unit tests
make MallocPlus_check
// To run the regression tests -- in the CLAMR directory
make clamr_clean_tests or rm -rf tests/testing
make clamr_tests
// To build the documentation
make doc
// or for a specific package documentation
make MallocPlus_doc
// To run on a Darwin MIC-NODE:
ssh darwin
salloc -p knc-mic
ssh [NODE]
// where [NODE] is $SLURM_NODELIST
// Load modules for build
module load compilers/intel/15.0.0
module load mpi/intelmpi-4.1.3.045-mic
module load cmake/2.8.11.1
/// These can be setup in the .login for the Intel MICs on darwin with
module load cmake/2.8.11.1 compilers/intel/15.0.0 mpi/intelmpi-4.1.3.045-mic
//The following source command can also be put in the .login with the first version for
// csh and the second for sh. When these are needed and what they do is still being studied
source /projects/opt/intel/compilers/composer_xe_2015.0.090/bin/compilervars.csh intel64
source /projects/opt/intel/compilers/composer_xe_2015.0.090/bin/compilervars.sh intel64
// For building on the host, change the module load to drop the -mic on the mpi
module load cmake/2.8.11.1 compilers/intel/15.0.0 mpi/intelmpi-4.1.3.045
// Build CLAMR on MIC:
cd CLAMR
cmake -DMIC_NATIVE=yes
make
// To execute:
ssh [NODE]-mic0
// Execute clamr_openmponly
export OMP_NUM_THREADS=120
In the CLAMR directory
cd CLAMR
./clamr_openmponly
For Cielito (Cray XE6) architecture
// Note that CLAMR has some problems with OpenMPI, but works at least with this version
// of mpich
module swap PrgEnv-pgi PrgEnv-gnu
module load cray-mpich2/6.3.1
module load cmake/2.8.9
cmake -DCMAKE_BUILD_TYPE=release -DCMAKE_CXX_COMPILER=`which CC` -DCMAKE_C_COMPILER=`which cc` .
or the equivalent (graphics and precision types below are currently the defaults)
cmake -DCMAKE_BUILD_TYPE=release -DGRAPHICS_TYPE=None -DPRECISION_TYPE=full_precision -DCMAKE_CXX_COMPILER=`which CC` -DCMAKE_C_COMPILER=`which cc` .
// To build
make
// There will be complaints about shared libraries. Go to each directory where it complains and
// edit the CMakeLists.txt and change "SHARED" to "STATIC"
vi genmalloc/CMakeLists.txt
vi graphics/CMakeLists.txt
vi memstats/CMakeLists.txt
vi mesh/hsfc/CMakeLists.txt
vi mesh/kdtree/CMakeLists.txt
vi mesh/zorder/CMakeLists.txt
vi s7/CMakeLists.txt
vi timer/CMakeLists.txt
// To run
aprun -n 128 -N 16 ../clamr_mpionly -n 128 -t 1000
// Watch-out for memory leaks with open-mpi 1.8.5 on GPU/MPI versions
// Set MPEHOME to location of MPE install or MPE_INCLUDE_DIR and MPE_LIBRARIES to
// the include path and the location of the MPE libraries
//
// need to edit configure.ac (b/c using autoconf package)
//
make // builds the executable
make install // copy the executable to destination
make clean // restores to the pre-make step
make distclean // restores to the pre-configure step
make installcheck // not always there
make dist // builds new distro into a new tar file
//
// need to edit Makefile.am (b/c using automake package)
//
/////////////////////
// Environment Setup
/////////////////////
// Update .bashrc by setting editor and loading modules
export EDITOR=vim
module load cuda
module load openmpi-gcc/1.4.1-64
/////////////////////
// Darwin
/////////////////////
To build on Darwin, The OpenCL libraries needed to build are only
available on back-end nodes. To get a node do:
Get a list of hardware partitions
% sinfo
For a generic tesla. The Kepler can be requested with the "K20" partition
% salloc -p tesla -N 1
% ssh $SLURM_NODELIST
The following modules should be loaded. This line can also be added
to the .login file to make it automatic
% module load cmake/2.8.10.2 mpi/openmpi-1.6.4-gcc-4.4.7
You must use a special version of GCC for CUDA that is not the default
compiler on Darwin. To use the correct compiler do:
% export PATH=/home/opt/gcc/4.4.6/bin:$PATH
Totalview on darwin
% module load totalview/8.11.0-2
% totalview <executable> -a <program command line>
Example
% totalview mpirun -a -n 2 ./clamr_mpionly -n 512 -l 2 -i 100 -t 10000
////////////////
// Moonlight
////////////////
%ssh -Y ml-fey
Get a backend node. The -nn is the number of nodes and the -np is the number
of processors per node
%llogin -nn 1 -np 8
Load modules
% module load gcc/4.7.2 openmpi/1.6.3 opencl/1.1
Add the following to the .login and you can select the gcc or intel compiler with
source ~/.login intel (for intel) or source ~/.login (for gcc)
========== .login file ============
if ("$1" =~ "intel") then
module load intel/13.0.1
else
module load gcc/4.7.2
endif
module load openmpi/1.6.3 opencl/1.1
===================================
********************* graphics and checkpoint restart ****************************
Just a little information on the movie graphics and the checkpoint/restart
The image and movie capabilities require the python imaging library (PIL or pillow
on Mac OSx) and mplayer (mencode comes with mplayer), respectively. An alternate
graphics output is done using the MagickWand library in many different formats.
For MAC:
brew install Homebrew/Python/pillow
brew install mplayer
For the MagickWand graphics library in ImageMagick
brew install imagemagick
************************************* clamr *************************************
This is all the code I adjusted in CLAMR. Below are notes on the new flags I added.
-R: Restart CLAMR from a specified backup file, so you will need to pass the
backup filename along with the path (EX: -R "checkpoint_output/backup00100.crx")
-b: The number of rollback files to maintain. Disk files are maintained with the
specified number of symlinks. In-memory images are limited to the specified
number of rollback images
-c: Backup CLAMR states to disk at the specified cycle intervals. They will be
put in the checkpoint_output directory
-C: Backup CLAMR states in-memory disk at the specified cycle intervals
-g: This generates the graphics files at the specified cycle intervals that are passed
in by this flag. The graphics data files .data & .lin will be dumped in the directory
where the executable is running.
-G: graphics file type. Possible values include none, data, bmp, gif, jpeg, mpeg, pdf,
png, and svg. Example -g 100 -G png will dump a graph#.png file in the graphics_output
directory every 100 cycles.
-U: Upper bound for what is the allowable mass percentage difference. This number can
be in integer, decimal, or scientific notation (EX: 1.23e-13).
************************************* scripts ************************************
This is the code for generating the images and movies. The generate_image.py,
generate_movie.sh, and FreeSansBold.ttf should be placed just outside of the graphics_output
directory. They will go into this directory to generate all image and movie data. The
movies that are generated are avi files, but they can be converted to any format using
on-line video converters. I did this, because the best quality for the movies is
generated by first making a avi file. The on-line movie converter I used was at
video.online-convert.com. Mplayer can be used in Linux to just play the movie in an
X-Terminal if you pass it the avi file.
FreeSansBold.tff: This is just the text file used by the python script to create the
lettering for number of iterations and time steps in the images. This file,
generate_image.py, and generate_movie.sh are all needed to create the movies. They
all just need to be in the same directory as the grahpics_info file that contains
the graphics information from the simulation. If you just want to create the images
and not the movie, then only this file and generate_image.py are needed in the
same directory as the graphics_output directory.
generate_image.py: This is python script that is used to generate the images from the
CLAMR simulation. The image data needs to be a directory called graphics_output. The
python script will go into this directory and generate the png files as well as an
text file called image list, which is used to generate a movie. You can pass the y
flag to this script if you want to have grid lines in the images as well. All image
data is saved in the .data files and all gridlines data is saved in the .lin files.
The FreeSansBold.ttf is needed to create the text in each image for the Iteration
number and Time Step values.
generate_movie.sh: This script uses the python code above to first generate the images,
and then generate a movie out the images. This script has a three flags that can be
used. They are explained below. The movie that is generated is dumped in the current
directory with the label test.avi
-y: Defines whether you want gridlines in the images for the movie generated
(False by default)
-f: How many FPS you want the movie to be (6 by default)
-s: What size do you want the images in the movie to be (800x800 by default).
So hopefully this all makes sense. I might have left something out. If you have any problems or
issues, just email me at [email protected]. I will respond ASAP.