Run Scripts: More Examples

Hello Container

There are different ways to execute commands in bueno run scripts. The following illustrates the most straightforward way to execute commands targeting both the container and the host shell (bash-like) emulator.

from bueno.public import container
from bueno.public import experiment
from bueno.public import host


def main(argv):
    experiment.name('hello-container')
    container.run('echo "hello from a container!"')
    host.run('echo "hello from the host!"')

This script can be executed as follows:

bueno run -a none -p hello-container.py

For simplicity, this particular invocation uses the none image activator. Please note that the underlying container target in this example is no more than a pass-through to the host and therefore equivalent to the direct host.run() invocation. We will detail how to change this behavior in an upcoming example.


Custom Actions

In this example, we consider a straightforward bueno run script with both pre- and post-experiment actions. These custom actions are implemented as separate, user-defined methods provided as callback functions to bueno that are automatically invoked at appropriate times before and after experiment execution, respectively.

Setup

As the name implies, a pre-action occurs before the application is executed. This action provides an opportunity to perform any setup needed to run the application.

Gathering information

In the definition for our post-action callback function, you will find that the variable number of arguments passed in kwargs is broken down into specific, meaningful values. They are:

cmd = kwargs.pop('command')    # The command used to execute the experiment.
out = kwargs.pop('output')     # The captured output (new-line delimited).
stm = kwargs.pop('start_time') # The start time of the provided command.
etm = kwargs.pop('end_time')   # The end time of the provided command.
tet = kwargs.pop('exectime')   # The execution time (in seconds) of the command.

In this example, cmd is the command string that was issued to the terminal emulator to run our bash script. out is a new-line delimited list containing stdout and stderr text emitted from our script. The last three variables contain timing information gathered from the execution of our test application. All of these values are used within the post_action scope to record experiment-specific data.


Data

This is an overview of some of the data acquisition and recording tools provided by bueno. This example will record the executed application’s output and some additional information about the system on which it was executed.

Exploring the code:

The following lines in the example run script setup two additional data assets. The first is called some-data.txt. Because there is no information assigned to this asset, it will simply generate an empty file. The file will be written to a subdirectory, illustrating that the relative location and depth of saved data are also user-definable.

The second asset is based on a run-time populated dictionary, and will contain information acquired about the user executing the script and the host on which it was run.

logger.log('Adding a file asset...')
# Add an arbitrary data file to a subdirectory.
data.add_asset(
    data.FileAsset('some-data.txt', 'subdir-a/subdir-b')
)

logger.log('Adding a YAML dictionary asset...')
adict = {}
# Populate the dictionary with relevant data.
adict['Application'] = {'argv': argv}
adict['System'] = {
    'whoami': host.whoami(),
    'hostname': host.hostname()
}
# Save the data to a file.
data.add_asset(
    data.YAMLDictAsset(adict, 'yaml-data')
)

Trying it Out

To test this for yourself, execute the following command:

bueno run -a none -p data.py

In the terminal output, you will find notes about the kind of information gathered and where it was saved. Several files are generated each time the run script is executed. Information about the host environment can be found in environment.yaml. For example,

Host:
  hostname: localhost.localdomain
  kernel: Linux
  kernel_release: 5.6.6-300.fc32.x86_64
  os_release: Fedora 32 (Workstation Edition)
  whoami: user

the run configuration stored in run.yaml:

Configuration:
  do_not_stage: false
  extras: null
  image: null
  image_activator: none
  output_path: /home/user/bueno/examples/data

and you will find that the second asset defined in the run script created a file with a similar format to the others. Additionally, there is a record of the run script executed and the output sent to the terminal at run-time.

The some-data.txt file asset is also present two levels down in a subdirectory. This illustrates how bueno supports the creation of any number of data assets as well as quite a few formats. Please consult this for a full list.


Extras

This example demonstrates how to use bueno’s --extras feature when executing a bueno run script. Additionally, it demonstrates how bueno utilities can test the validity of import statements to give meaningful warnings instead of simply terminating with a run-time error.

Try to run this example, execute the following:

bueno run -a none -p extras.py

When reviewing the output, you will find there are warnings emitted during its execution:

*** Note: mymod is not imported ***
*** Note: mypackmod not imported ***

This shows that without using the --extras feature in this particular case we have not correctly imported necessary libraries. Now, try executing the run-example script instead:

./run-example

Please notice that the prior warnings have been replaced with hello statements from the newly imported library. Examining the contents of the run-example script, you will find that it is a modified version of the bueno run command that we have been using for the last few examples:

bueno run -a none --extras .:./mypackage -p ./extras.py


Build And Run

Build

Previous examples used only bueno’s none image activator. This example, however, explores using bueno’s charliecloud image activator. We will start by building a container image using Charliecloud for nbody, our example MPI application.

ch-build2dir --force -t nbody-img -f ./Dockerfile.mpich . .

Once completed, there will be a new directory named nbody-img in your working directory. We will use the contents of this directory when executing the bueno run script.

Run

Instead of simply instructing bueno to execute our run script as we have in the past, we will provide bueno with a container image to work with.

bueno run --do-not-stage -i nbody-img -p build-and-run.py

If successful, the terminal will fill with output from the containerized application. If you encountered some errors, try the workarounds outlined in the following tip:

Tip: If you encountered an error similar to this:

What: run error encountered.
Why:  Cannot determine the number of nodes in your job.

or this:

ch-run[251876]: join: no valid peer group size found (ch-run.c:382)

Try the following:

export SLURM_CPUS_ON_NODE=2 # Or another number to match system resources.

Run Scripts: Real-World Examples

For more information on bueno’s use in practice, including source code and online documentation, please consult the information linked below: