Advanced Usage

Command and Job Output

When BEE submits steps of a workflow, by default the output from the underlying batch scheduler (Slurm, LSF, etc.) will be saved by BEE. This includes all output from other helping programs, such as the container runtime and the batch scheduler itself. Sometimes this output can be useful when diagnosing runtime failures. In most cases you shouldn’t need to use this unless the error is outside of your program (this often includes environment-related and setup/installation issues). Note, to allow for better provenance, this output is also saved when archiving a workflow. This is also one reason why the specific path below should not be changed unless you know what you’re doing.

By default, output from job, if not captured by an stdout or an stderr option within your workflow CWL will be directed to $bee_workdir/workflows/$workflow_id/$task_name-$task_id, with the meaning of each component listed below:

$bee_workdir

BEE’s workdir, which is set in the bee.conf file and by default is $HOME/.beeflow

$workflow_id

The same ID used to start the workflow

$task_id

A randomly generated UUID which BEE uses to keep track of the task

$task_name

The task/step name used within the workflow submission

Within this directory, you can find the submission script and two output files for stderr (extension .err) and stdout (extension .out).

If something bad happens, such as if BEE fails unexpectedly, then you may need to examine these output files. Sometimes the error may have to do with the environment or with a task’s requirements. If you’re unable to find the cause of the problem, then you should contact the BEE developers.

Multiple Workflows

BEE allows orchestration of multiple workflows. You can try running concurrent example workflows from cat-grep-tar workflow with two different names and different output paths. Using the same example is over simplified but does demonstrate running concurrent workflows. You may have requirements where you run the same workflow with different inputs and paths for outputs or other variations. You could prepare a workflow with different yaml files and make submissions for each specification. We suggest you use unique names for each workflow, but it isn’t necessary since BEE assigns a unique workflow ID to each one. For demonstration we show how you might run our simple example as two workflows with different names.

The procedure would be to submit some workflows like the following:

mkdir results1
mkdir results2
beeflow submit cgt1 ./cat-grep-tar.tgz workflow.cwl input.yml results1
beeflow submit cgt2 ./cat-grep-tar.tgz workflow.cwl input.yml results2
beeflow list

Name  ID      Status
cgt1  b33fd3  Pending
cgt2  9a378c  Pending

You could then start each workflow with the beeflow start <WFID> command and query for the status of their tasks separately using beeflow query <WFID>.