Troubleshooting¶
Failed simulations¶
If most or all of your simulations are failing (and generate messages like “Job init0 failed” or “Your simulations are failing to run”), troubleshooting is necessary at the level of the simulator (BioNetGen or libRoadRunner).
Check the simulation logs¶
Failed simulations will send their logs (generally stdout and stderr) to a FailedSimLogs folder in the specified output directory. These logs should usually contain more information about why the simulator failed to run.
By default, PyBNF saves logs of roughly the first 10 failed simulations encountered. If PyBNF is run with the -d flag, logs from all failed simulations will be saved. If the fit was run with delete_old_files=0 in the config file, all logs can be found in the appropriate folders in the Simulations/ directory.
For BNGsim-backed runs (BNGL network, NF, SBML, and Antimony), PyBNF additionally writes a structured failure report next to each failed simulation. The report captures the backend name (bngsim-net, bngsim-nf, bngsim-sbml, bngsim-antimony), the bngsim version, the model identity, the parameter set, the action/method/seed that was running, the exception type and message, the full traceback, and a path to the generated .net/XML/SBML/Antimony input. Like the legacy logs, it is collected into FailedSimLogs/ under the standard rules (first ~10 failures, or all failures with -d).
For BNGL simulations¶
Confirm that the BioNetGen path is set¶
Confirm that PyBNF is looking for BioNetGen in the right place: it will use the bng_command specified in your config file if present, and otherwise will use your BNGPATH environmental variable (we recommend this second option). To check that BNGPATH is set correctly, run $BNGPATH/BNG2.pl; you should see a help message including your BioNetGen version number. If not, try setting BNGPATH again.
Confirm that the model runs in BioNetGen¶
If the simulation logs are not sufficient to diagnose the problem, you may want to check whether you can run BioNetGen on the PyBNF-generated model files by hand. Run the fit with the config key delete_old_files=0, and refer to the subdirectory of the Simulations folder corresponding to a job that failed. Try running BioNetGen on that .bngl file and check for errors; also examine the .bngl file and confirm that PyBNF did not introduce any errors to the model.
If your model is not running in BioNetGen, the best place to find help is the documentation and troubleshooting for BioNetGen, at https://bionetgen.org/
- Known BioNetGen issues:
If you are using a Linux distribution other than Ubuntu, it may be necessary to compile BioNetGen from source rather than installing the pre-built binary. Specifically, on CentOS, the binary appears to work at first glance, but fails to parse models containing functions.
For SBML simulations¶
Confirm accuracy of SBML¶
If the SBML file was generated in COPASI, refer to Unexpected behavior when generating SBML files in COPASI.
CVODE errors¶
For SBML models, if your logs in FailedSimLogs/ include errors from CVODE such as “CV_ERR_FAILURE: Error test failures occurred too many times during one internal time step” or “CV_TOO_MUCH_WORK: The solver took mxstep internal steps but could not reach tout”, it means that CVODE, the ODE integrator that simulates SBML models on both sbml_backend settings (through libRoadRunner on roadrunner, through bngsim on bngsim), decided that the model was too difficult to simulate and gave up. This might happen when the solution to the ODE system is not sufficiently smooth.
It may be possible to run such simulations with a different SBML integrator, set with the sbml_integrator key.
Check the absolute tolerance first when sbml_backend = bngsim. Both of those errors are
what an atol mismatched to the model’s units looks like from the outside. CVODE weights each
state by rtol*|y| + atol, so an atol far beneath a species’ own magnitude holds the
integrator to an accuracy nobody asked for, and it pays in steps until it exhausts mxstep.
PyBNF derives atol from the model’s species magnitudes for exactly this reason, but the
derivation is only allowed to tighten by default, so a model whose species sit far above one
keeps a tolerance 5+ decades tighter than its own scale asks for. See
sbml_rtol for the full account; in order of what to try:
sbml_atol = auto— let the derivation loosen as well as tighten. Costs nothing on a model whose species sit at or below one, and is the first thing to try on one whose species are large.sbml_atol = <number>— state one tolerance for every species of every model, replacing the derivation entirely.atol:/rtol:/species_atol:on a single model’s model: declaration (edition >= 2) — the same settings scoped to one model of a multi-model fit, plus the only way to hand-write a tolerance for a named species. Use it when one model, or one species, is the one giving trouble.sbml_atol = tracking— an absolute tolerance that follows the trajectory rather than staying where the initial values put it, for a species that decays far below its own starting magnitude. It costs integrator steps; lower its depth if a model that integrated before stops.
Going the other way — a trajectory that looks wrong rather than one that fails — is the same
key: a model whose species sit below 1e-8 carries no significant digits at the backend
default, and its forward sensitivities carry fewer still.
Resource not available¶
We have seen this error message come up and cause all simulations to fail when some especially badly behaved SBML process is still running from a previous fitting run (see Jobs still running after PyBNF stops). Killing all of the offending processes typically resolves this error.
Timed out simulations¶
PyBNF enforces a maximum run time for simulations, with a default value of 1 hour. If you find a large number of your simulations are timing out, increase this value using the config key wall_time_sim.
A time limit is also enforced for network generation in BNGL models. The default value is 1 hour, and this can be modified with the wall_time_gen key.
Both of these bound one unit of work, not the fit. To bound the fit as a whole – for a fixed compute allocation, or so a run that is not converging still leaves usable results – set wall_time_fit (seconds). When it expires PyBNF stops launching work and writes its normal end-of-fit results for the best parameter set found so far, plus a Results/stop_reason.txt saying why it stopped. Killing the process externally writes none of that end-of-fit output; what survives is the run’s periodic checkpoint – Results/sorted_params_backup.txt and, for a likelihood objective, Results/information_criteria_backup.txt (see backup_information_criteria) – which is enough to score the best fit reached, but is a snapshot rather than the run’s stated result.
Which methods did my run actually execute?¶
Every run writes Results/method_chain.json: the method chain the config asked for, the
chain that ran, and one entry per phase (the fit, the refine, the bootstrap replicates)
with its stop reason, its completed simulations, and the best objective it reached:
{
"job_type": "cmaes",
"requested_methods": ["cmaes", "gntr"],
"executed_methods": ["cmaes", "gntr"],
"phases": [ ... ]
}
requested_methods longer than executed_methods means a requested phase did not run,
and the phase’s status and reason say why. The usual cause is a wall_time_fit
that the search consumed; see wall_time_refine_frac, which holds part of the budget
back for the refine.
Where did my fit actually start?¶
Every run writes Results/start_point.txt, before any parameter set is scored. One row per
free parameter, in declaration order (not the alphabetical order the parameter tables use),
giving the start value, where it came from, and the box it was checked against:
starts 5
starts_pinned 1
# parameter start source lower upper
kcat 17.5 start_point 0.01 1000
Km 50.5 box_center 1 100
The source column is the thing to read. start_point / initial_value mean you pinned
that coordinate; box_center means it came from the 0.5 quantile of the bounded prior;
sampled means it was drawn from the prior or bounds, as a population algorithm or sampler
does; refine means the previous phase’s best fit was injected by a method chain.
This matters most when a fit converges to a plausible-looking number from the wrong place, which
has no symptom of its own. In particular, box_center for a truncated non-uniform prior is
the prior’s median, and under asymmetric truncation the median is not the location parameter –
a normal prior with mean: 0.088, sd: 0.2 truncated to [0, 1] has its median at
0.173, a factor of ~2 away. If you meant to start at the mean, say so with
start_point rather than relying on the box centre.
starts and starts_pinned describe a multi-start fit: a declared start point pins start 0
and the remaining starts stay independent draws, so your point governed one of them, not all.
A refine writes Results/start_point_refine.txt beside it. A resumed run writes neither –
it leaves the original run’s record intact, since a resume has no start of its own.
Note
For models simulated with the bngsim backend, wall_time_sim is honored
in-process on every supported backend: ODE, SSA, PSA, NFsim/nf,
and RuleMonkey/rm. RuleMonkey polls a cooperative cancellation
callback every ~1024 SSA events, so an over-budget rm run is
stopped at the next polling boundary.
Should I believe my fit, or run it again with more starts?¶
A fit that runs several searches from different starting points reports the best of them, and that one number cannot be checked on its own. Two very different runs look identical in it: one where every start reached about the same objective value, and one where every start landed somewhere different and the reported answer is only the least bad of them.
So a fit with more than one start writes Results/multistart_summary.txt, one row per start
sorted by objective value from best to worst:
starts 20
best_objective 143.2085
worst_objective 891.4471
median_objective 143.2166
starts_near_best 17
# rank start objective iterations evaluations reason
1 7 143.2085 38 402 gradient is flat (‖v·Jᵀr‖∞ ≤ 1e-08)
2 2 143.2091 41 437 gradient is flat (‖v·Jᵀr‖∞ ≤ 1e-08)
...
20 13 891.4471 60 631 reached max_iterations (60)
Read the objective column downward. A long flat run of starts at the same low value means the
search found a consistent answer and more starts would very likely not change it. A staircase
with no flat section at the top means it did not, so run more starts, or take another look at
the model and at the parameter bounds. starts_near_best counts the flat run for you, and
a short version of all of this is printed at the end of the run.
Two rows deserve a second look wherever they appear. inf in the objective column is a start
that produced no usable fit at all, usually because its start point failed to simulate; if every
row says that, the parameter box is somewhere the model cannot be integrated. A reason of
“did not finish before the fit ended” or “did not start before the fit ended” means the run
stopped before that start was done with, which is what wall_time_fit expiring looks like
here.
This applies to trf, lbfgs, gntr, powell, sim, ms, the polishing phase
of profile_likelihood, and the metaheuristics de, ade, ss and pso. The
number of starts comes from population_size for the gradient methods and from n_starts
for the rest. A refine writes Results/multistart_summary_refine.txt; a fit with a single
start writes nothing, since there would be nothing to compare it against.
My stochastic fit reports an objective value I cannot reproduce¶
When a model is stochastic, such as one simulated with the stochastic simulation algorithm or with NFsim, running it twice with the same parameter values gives two different answers. The objective value PyBNF computes for a parameter set is therefore a noisy measurement of it rather than a fixed number.
A fit picks its answer by taking the best objective value it ever saw, and each of those values came from a single simulation. A long fit scores tens of thousands of parameter sets, so the winner of that comparison is very often the parameter set that happened to get a lucky simulation rather than the parameter set that is genuinely best. Two things then come out wrong. The reported objective value is the best of many noisy draws, so it is optimistic, sometimes by several times the spread among the good parameter sets. And the reported parameter values are not the best ones found, because a slightly worse parameter set with a lucky draw beats a better one with an average draw.
So a fit that uses at least one stochastic model ends with one more stage. The top
best_fit_candidates parameter sets are each run best_fit_replicates more times, they
are ranked by their average objective value, and the winner of that ranking is what the run
reports as its best fit. All of those simulations are submitted at once, which uses
processors that would otherwise sit idle at the end of a run. The stage writes
Results/best_fit_confirmation.txt:
candidates 3
replicates_requested 4
winner iter2p0h4
winner_mean_objective 72838.74908
winner_standard_error 5521.750431
winner_search_objective 54877.73818
optimism 17961.0109
# rank name mean_objective standard_error std_deviation runs failed search_objective
1 iter2p0h4 72838.74908 5521.750431 11043.50086 4 0 54877.73818
2 init4 87262.13928 3134.119241 6268.238482 4 0 99030.06578
3 iter2p5h3 100953.6321 3221.85557 6443.71114 4 0 101437.2518
mean_objective is the number to quote. Compare it against search_objective in the
same row to see how much of the search’s answer was luck; the optimism line does that
subtraction for the winner. standard_error says how well this stage told the candidates
apart, so two rows whose averages differ by less than their standard errors have not really
been separated and need more replicates. A search_rank line appears when the winner is
not the parameter set the search itself would have reported.
The stage runs under edition = 2 and above, where it defaults to ten candidates at ten
replicates each. Under the legacy edition it is off, because an unchanged configuration file
has to keep behaving as it always has; a legacy fit with a stochastic model prints a warning
saying so, and turns the stage on by setting the two keys itself. Setting
best_fit_replicates = 0 turns it off anywhere. Nothing happens for a fit with no
stochastic model, and no fitting method searches any differently either way.
A run whose wall_time_fit budget is already spent skips the stage and says so, since a
budget is a promise about the whole run and this costs more simulations. It is also skipped,
with a warning at startup, when stochastic_seed is one of the _honorbngl modes and
every stochastic model in the fit carries an explicit seed argument, because then every
replicate would reproduce the same trajectory and there would be nothing to average.
This covers the answer the fit reports, and not the search that produced it. The same noise also affects the search while it is running, where a lucky value recorded for a population member can never be displaced except by a luckier one. That is a larger piece of work, and the table above is a way to measure how much it matters for your model before deciding whether to worry about it.
smoothing is a different setting for a related problem. It runs every evaluation of the
whole fit several times and averages the simulated curves, which multiplies the cost of the
entire fit, and it still leaves the final choice being made among the best of many noisy
values. The two work together: with smoothing on, each of the replicate runs here is a
whole smoothed evaluation, so the cost of this stage is candidates times replicates times
smoothing simulations.
Unexpected behavior when generating SBML files in COPASI¶
While COPASI is a useful tool for generating SBML files, it is important to note that some settings in COPASI do not get converted into SBML. This can lead to unexpected model behavior in PyBNF.
To help confirm that your model is running as expected, you can set delete_old_files=0 in your config file, which causes the model output as it was simulated by libRoadRunner in PyBNF to be saved in the Simulations/ directory.
The following are known issues in translating from COPASI to SBML / libRoadRunner:
Writing formulas in terms of derivatives of species is possible in COPASI, but does not export to SBML.
If you rename a parameter or species in COPASI (some time after its creation), the parameter / species is not renamed in the exported SBML, likely causing a PyBNF error about a name not being found. To effectively rename a parameter or species, do a find/replace for
id="oldname"in the SBML file itself, or delete the object in COPASI and create a new one.Defining an “Initial expression” for the concentration of a species is supported in COPASI, but does not export to SBML.
Events are handled differently if the trigger is true at time 0. COPASI provides options for behavior, with the default being that the event does not fire. These options do not export to SBML. In libRoadRunner and PyBNF, the only option is that the event continues to fire as long as the trigger remains true. Note that this is different behavior than for events triggered at time > 0, which will only fire once.
Too many open files¶
Some highly parallelized runs may encounter the error “Too many open files”. This error occurs when PyBNF exceeds the number of open files allowed by the system for a single program. When this error comes up, it prevents PyBNF from saving results and backups of the run, and may also interfere with its ability to run simulations.
Source of the bug: Each time that PyBNF submits a job, it uses 2 file handles to keep track of the connection between the scheduler and the worker. These file handles are closed eventually, but remain open for a short time after a job completes. If you have a fast running simulation, you might get ~ 5 iterations’ worth of these handles left open at the same time. If that many handles exceeds your system limit, you will encounter this bug.
Remedies: You can check the limit of open files per program on the command line: ulimit -n gives you the “soft” limit, and ulimit -Hn gives you the “hard” limit. The soft limit is what is actually enforced. You can increase the soft limit up to the hard limit with, for example ulimit -n 4096 if your hard limit is 4096 (this only affects the current terminal, so do it in the same terminal where you will run PyBNF). This might give you enough file handles to avoid the bug. If not, the hard limit can be increased with root access to the machine.
If you are unable to increase the open file handle limit, then you will have to reduce the number of parallel jobs submitted in PyBNF by adjusting the parallel_count or population_size settings.
Too many threads¶
This error can come up in parallelized runs in which simulations are very fast. Similar to the Too many open files error, it occurs when PyBNF exceeds the number of threads allowed by the system for a single user.
You can check the thread limit on the command line with ulimit -u. Many operating systems have this limit very high (over 100,000), but if yours has it set on the order of 4096, it could cause this error.
We recommend having an administrator with root access increase your default thread limit on the machine. Edit the file /etc/security/limits.conf and add the lines:
username soft nproc 100000
username hard nproc 100000
where username is your user name, and 100000 is the new thread limit (use any reasonably large value). Restart the system for the changes to take effect.
We do not recommend increasing the thread limit via the command line as in Too many open files: This change would only affect the current terminal, so although PyBNF could keep running, the rest of your system would become unresponsive after the original limit was exceeded.
Jobs still running after PyBNF stops¶
Ordinarily, PyBNF kills simulation jobs that run longer than the time limit. However, if PyBNF itself exits (terminated by the user, or finished a fitting run with jobs still pending), then it is no longer able to enforce the time limit on any jobs that are still running. Any such jobs will continue until they finish or are killed.
If the undead jobs become problematic, it is possible to kill them manually. Use the command top to see if you have any such jobs: the processes will have the name run_network, NFsim, or python, depending on which simulator you are using. Note the PID of the offending process(es), and then run kill <PID> on the appropriate PIDs. It is also possible to kill all of the jobs at once by running killall run_network, killall NFsim, or killall python, provided that you have no running processes of the same name that you want to keep.
My fitted noise scale is missing from sorted_params_final.txt¶
You set noise_profiling = 1, so that scale is no longer
searched: PyBNF solves for it analytically at every evaluation instead of proposing values
for it. It is therefore not a coordinate of the best parameter set and appears in no
sorted_params_*.txt row.
Its fitted value is in Results/profiled_noise.txt (and is echoed on the console at the end
of the run). It is still an estimated parameter, so it is still counted in k in
Results/information_criteria.txt. Drop noise_profiling if you want the scale searched
as an ordinary free parameter and reported alongside the model parameters.
Could not start the workers on the other machines¶
dask ssh, which PyBNF runs to start the workers of a multi-machine run, exited before any
worker started, so the run stops about ten seconds in. Everything dask ssh said is quoted
in the message and repeated in the log file.
The most common cause is the login itself. dask ssh does not run your ssh command: it
logs in with the paramiko library, which can offer a public key or a typed password and nothing
else. A cluster whose nodes authenticate to each other by host-based or Kerberos (GSSAPI) SSH
therefore refuses it however you configure it – on a machine where ssh othernode hostname
from the same shell succeeds – and creating SSH keys cannot fix it, because the cluster is not
asking for a key. Which ways of starting a run log in to other machines gives
a one-line test of the login PyBNF actually makes.
Two ways of running on several machines need no login at all, and neither is affected:
starting the workers with srun (-t slurm-srun), inside the allocation SLURM
already granted, and starting the scheduler and workers yourself and giving
PyBNF the scheduler file with -s.
PyBNF has encountered a fatal error¶
This error occurs when the scheduler loses connection with the cluster. The simulation data is generally backed up and the simulation can be resumed from the point it exited using the -r flag ‘pybnf -c .conf -r’.
An unknown error occurred¶
If you get this message, you found an error that we did not catch during development. Sorry. It might be an unusual, user-generated situation that we didn’t think of but is fixable on your end, or could be a bug in the PyBNF source code.
Refer to the log file to try to diagnose the problem - it will contain the Python traceback of the error that was thrown, which sometimes contains enough information to identify what happened.
Rerun the fit with the debugging -d flag to generate a more detailed log file (with a “debug” tag).
If you would like to report the bug to the developers ( https://github.com/lanl/PyBNF/ ), it will be helpful for us if you include the debug log file with your bug report.
Other issues¶
If you encounter a bug that is not documented here, or have a request for a new feature, please contact the developers at https://github.com/lanl/PyBNF/ .