Building
Obtaining the Source Code
phoebus
uses submodules extensively. To make sure you get them all,
clone it as
git clone --recursive git@github.com:lanl/phoebus.git
or as
git clone git@github.com:lanl/phoebus.git
cd phoebus
git submodule update --init --recursive
Prerequisites
To build phoebus
, you need to create a build directly, as in-source builds are not supported.
After cloning the repository,
cd phoebus
mkdir bin
cd bin
Example builds
Below are some example build commands
MPI-parallel only
The following will build phoebus
with MPI parallelism but no shared
memory parallelism
cmake ..
make -j
OpenMP-parallel
The following will build phoebus
with OpenMP parallelism only
cmake -DPHOEBUS_ENABLE_MPI=Off -DPHOEBUS_ENABLE_OPENMP=ON ..
make -j
Cuda
The following will build phoebus
with no MPI or OpenMP parallelism.
cmake -DPHOEBUS_ENABLE_CUDA=On -DCMAKE_CXX_COMPILER=${HOME}/phoebus/external/singularity-eos/utils/kokkos/bin/nvcc_wrapper -DKokkos_ARCH_HSW=ON -DKokkos_ARCH_VOLTA70=ON -DPHOEBUS_ENABLE_MPI=OFF ..
A few notes for this one: - Note here the -DCMAKE_CXX_COMPILER
flag.
This is necessary. You must set the compiler to nvcc_wrapper
provided by Kokkos. - Note the -DKokkos_ARCH_*
flags. Those set the
host and device microarchitectures and are required. The choice here is
on an x86_64
machine with a volta
GPU.
Build Options
The build options explicitly provided by phoebus
are:
Option |
Default |
Comment |
---|---|---|
PHOEBUS_ENABLE_CUDA |
OFF |
Cuda |
PHOEBUS_ENABLE_HDF5 |
ON |
HDF5. Required for output and restarts. |
PHOEBUS_ENABLE_MPI |
ON |
MPI. Required for distributed memory parallelism. |
PHOEBUS_ENABLE_OPENMP |
OFF |
OpenMP. Required for shared memory parallelism. |
MACHINE_CFG |
None |
Machine-specific config file, optional. |
Some relevant settings from Parthenon and Kokkos you may need to play with are:
Option |
Default |
Comment |
---|---|---|
CMAKE_CXX_COMPILER |
None |
Must be set to |
Kokkos_ARCH_XXXX |
OFF |
You must set the GPU architecture when compiling for Cuda |
You can see all the Parthenon build options here and all the Kokkos build options here
Cmake machine configs
If you are proficient with cmake
You can optionally write a
cmake
file that sets the configure parameters that you like on a
given machine. Both phoebus
and parthenon
can make use of it.
You can point to the file with
-DMACHINE_CFG=path/to/machine/file
at config time or by setting the environment variable MACHINE_CFG
to
point at it, e.g.,
export MACHINE_CFG=path/to/machine/file
An example machine file might look like
# Machine file for x86_64-volta on Darwin
message(STATUS "Loading machine configuration for Darwin x86-volta node")
message(STATUS "Assumes: module load module load gcc/7.4.0 cuda/10.2 openmpi/4.0.3-gcc_7.4.0 anaconda/Anaconda3.2019.10 cmake && spack load hdf5")
message(STATUS "Also assumes you have a valid spack installation loaded.")
set(PHOEBUS_ENABLE_CUDA ON CACHE BOOL "Cuda backend")
set(PHOEBUS_ENABLE_MPI OFF CACHE BOOL "No MPI")
set(Kokkos_ARCH_HSW ON CACHE BOOL "Haswell target")
set(Kokkos_ARCH_VOLTA70 ON CACHE BOOL "volta target")
set(CMAKE_CXX_COMPILER /home/jonahm/phoebus/external/parthenon/external/Kokkos/bin/nvcc_wrapper CACHE STRING "nvcc wrapper")
you could then configure and compile as
cmake -DMACHINE_CFG=path/to/machine/file ..
make -j
Running
Run phoebus from the build
directory as
./src/phoebus -i path/to/input/file.pin
The input files are in phoebus/inputs/*
. There’s typically one input
file per problem setup file.
Submodules
parthenon
asynchronous tasking and block-AMR infrastructuresingularity-eos
provides performance-portable equations of state and PTE solverssingularity-opac
provides performance-portable opacities and emissivitiesKokkos
provides performance portable shared-memory parallelism. It allows our loops to be CUDA, OpenMP, or something else. By default we use theKokkos
shipped withparthenon
.
External (Required)
cmake
for building
Optional
hdf5
for output (must be parallel if MPI is enabled)MPI
for distributed memory parallelismpython3
for visualization