Building

Obtaining the Source Code

phoebus uses submodules extensively. To make sure you get them all, clone it as

git clone --recursive git@github.com:lanl/phoebus.git

or as

git clone git@github.com:lanl/phoebus.git
cd phoebus
git submodule update --init --recursive

Prerequisites

To build phoebus, you need to create a build directly, as in-source builds are not supported. After cloning the repository,

cd phoebus
mkdir bin
cd bin

Example builds

Below are some example build commands

MPI-parallel only

The following will build phoebus with MPI parallelism but no shared memory parallelism

cmake ..
make -j

OpenMP-parallel

The following will build phoebus with OpenMP parallelism only

cmake -DPHOEBUS_ENABLE_MPI=Off -DPHOEBUS_ENABLE_OPENMP=ON ..
make -j

Cuda

The following will build phoebus with no MPI or OpenMP parallelism.

cmake -DPHOEBUS_ENABLE_CUDA=On -DCMAKE_CXX_COMPILER=${HOME}/phoebus/external/singularity-eos/utils/kokkos/bin/nvcc_wrapper -DKokkos_ARCH_HSW=ON -DKokkos_ARCH_VOLTA70=ON -DPHOEBUS_ENABLE_MPI=OFF ..

A few notes for this one: - Note here the -DCMAKE_CXX_COMPILER flag. This is necessary. You must set the compiler to nvcc_wrapper provided by Kokkos. - Note the -DKokkos_ARCH_* flags. Those set the host and device microarchitectures and are required. The choice here is on an x86_64 machine with a volta GPU.

Build Options

The build options explicitly provided by phoebus are:

Option	Default	Comment
PHOEBUS_ENABLE_CUDA	OFF	Cuda
PHOEBUS_ENABLE_HDF5	ON	HDF5. Required for output and restarts.
PHOEBUS_ENABLE_MPI	ON	MPI. Required for distributed memory parallelism.
PHOEBUS_ENABLE_OPENMP	OFF	OpenMP. Required for shared memory parallelism.
MACHINE_CFG	None	Machine-specific config file, optional.

Some relevant settings from Parthenon and Kokkos you may need to play with are:

Option	Default	Comment
CMAKE_CXX_COMPILER	None	Must be set to nvcc_wrapper with cuda backend
Kokkos_ARCH_XXXX	OFF	You must set the GPU architecture when compiling for Cuda

You can see all the Parthenon build options here and all the Kokkos build options here

Cmake machine configs

If you are proficient with cmake You can optionally write a cmake file that sets the configure parameters that you like on a given machine. Both phoebus and parthenon can make use of it. You can point to the file with

-DMACHINE_CFG=path/to/machine/file

at config time or by setting the environment variable MACHINE_CFG to point at it, e.g.,

export MACHINE_CFG=path/to/machine/file

An example machine file might look like

# Machine file for x86_64-volta on Darwin
message(STATUS "Loading machine configuration for Darwin x86-volta node")
message(STATUS "Assumes: module load module load gcc/7.4.0 cuda/10.2 openmpi/4.0.3-gcc_7.4.0 anaconda/Anaconda3.2019.10 cmake && spack load hdf5")
message(STATUS "Also assumes you have a valid spack installation loaded.")

set(PHOEBUS_ENABLE_CUDA ON CACHE BOOL "Cuda backend")
set(PHOEBUS_ENABLE_MPI OFF CACHE BOOL "No MPI")
set(Kokkos_ARCH_HSW ON CACHE BOOL "Haswell target")
set(Kokkos_ARCH_VOLTA70 ON CACHE BOOL "volta target")
set(CMAKE_CXX_COMPILER /home/jonahm/phoebus/external/parthenon/external/Kokkos/bin/nvcc_wrapper CACHE STRING "nvcc wrapper")

you could then configure and compile as

cmake -DMACHINE_CFG=path/to/machine/file ..
make -j

Running

Run phoebus from the build directory as

./src/phoebus -i path/to/input/file.pin

The input files are in phoebus/inputs/*. There’s typically one input file per problem setup file.

Submodules

• parthenon asynchronous tasking and block-AMR infrastructure

• singularity-eos provides performance-portable equations of state and PTE solvers

• singularity-opac provides performance-portable opacities and emissivities

• Kokkos provides performance portable shared-memory parallelism. It allows our loops to be CUDA, OpenMP, or something else. By default we use the Kokkos shipped with parthenon.

External (Required)

• cmake for building

Optional

• hdf5 for output (must be parallel if MPI is enabled)

• MPI for distributed memory parallelism

• python3 for visualization