PyBNF experimental design (pybnf.design)

The pybnf.design package works out which measurement to make next. It reads the expected Fisher information pybnf.gradient.assemble_fisher_hessian() already builds for the gntr optimizer, and rests on one fact about it: the information is a plain sum over the measured points, so the information a planned measurement would add is that measurement’s own term (pybnf.gradient.iter_fisher_points()).

It has four parts: candidates, which enumerates the measurements a design may choose from and scores each by handing the Fisher assembly a one-row dataset holding the model’s own prediction at that point; criteria, which reduces an information matrix to the single number two designs are compared on; greedy, which chooses the measurements one at a time; and report, which writes the recommendation together with the confidence intervals it is expected to produce.

The user-facing account – what a design may recommend, the criteria, and the grid controls that let it propose a time you have never measured – is in Experimental design (what to measure next).

Configuration

The configuration keys an experimental design reads (#574).

These live here, in the design package, rather than beside one method, because two job types read the same keys: job_type = design runs a design on its own, and job_type = profile_likelihood can end by recommending one. Both schemas inherit this class, so the keys mean the same thing and are documented once (ADR-0006 co-locates a method’s own keys with the method; a set of keys shared by two methods has to sit somewhere both can see).

class pybnf.design.config.DesignFields(*, design_points: int = 5, design_criterion: str = 'a', design_target: Any = None, design_observables: Any = None, design_confidence: float = 0.95, design_grid: int = 0, design_t_end: float = 0.0)

Optimal experimental design settings, shared by job_type = design and the design report a profile_likelihood run can write.

design_points is how many measurements to recommend. The same measurement may be recommended more than once, which means measure it that many times.

design_criterion is what makes one design better than another: a for the average variance of the parameters (the default, and the classical c-criterion when design_target names a single parameter), d for the volume of the joint confidence region, e for the worst-determined direction.

design_target names the parameters the design is aimed at. Absent, it aims at all of them. Only the A-criterion can use it; the other two are properties of the whole information matrix.

design_observables restricts the candidate measurements to a named set of observables, for when only some assays can actually be run. Absent, every observable the fit already measures is a candidate.

design_confidence is the confidence level of the predicted intervals in the report. It has the same meaning as profile_likelihood_confidence, and a profile_likelihood run that writes a design report uses that key instead so the two halves of its output agree.

design_grid and design_t_end widen what the design is allowed to recommend. A design can only propose a time the model is already simulated at, and for a time course PyBNF simulates the times the data was measured at, so by default the only new measurement it can propose is a repeat of an existing one. design_grid adds that many extra simulated times, spread evenly from the first measurement out to design_t_end (which defaults to the last measurement). Set both to let a design say “measure at a time you have never measured”, which is usually the point of asking.

model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'extra': 'forbid', 'populate_by_name': True, 'validate_by_alias': True, 'validate_by_name': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

Candidates

Enumerating the measurements a design may choose from, and what each one would tell you (#574).

A candidate measurement is one observable, in one experiment, at one time. Two rules keep the candidate space honest and cheap:

• The observable has to be one that experiment already measures. Then its noise model is known, because it is the one the fit is already using. Proposing an observable that has never been measured would mean inventing a precision for an assay nobody has run, and the answer would depend entirely on that invented number.

• The time has to be one the simulation already passes through. The sensitivities at every simulated time were computed when the best fit was scored, so every candidate is free: no model is re-solved to enumerate or to score the candidate space.

Scoring one candidate means asking what the information matrix would gain if that point were measured. PyBNF can already answer that, because the information is a sum over measured points and iter_fisher_points() yields the points one at a time. So a candidate is scored by handing the machinery a planned measurement: a one-row dataset at the candidate time, whose value is the model’s own prediction at the best fit. That is what the expected Fisher information means – the information you expect from data generated by the fitted model – and it is why every noise model comes along for free. A noise scale read from a data column takes the value from the nearest real measurement of that same observable, which assumes the planned measurement is as precise as the ones already made; a scale that is a fitted parameter, a constant, or a function of the prediction needs no assumption at all.

class pybnf.design.candidates.CandidateMeasurement(model: str, experiment: str, observable: str, time: float, independent_variable: str = 'time')

One measurement a design may recommend: an observable, in an experiment, at a time.

independent_variable is that experiment’s own name for its first column, so a report can say time = 4.5 for a time course and use the right word for anything else.

class pybnf.design.candidates.CandidateSet(measurements: list = <factory>, blocks: list = <factory>)

Every candidate measurement, with the information matrix each one would add.

blocks[i] is the (n_param, n_param) matrix candidate measurements[i] contributes, in sampling space, ready to be added to a baseline information matrix.

total(n_param)

The information of measuring everything at once – the most any design over this candidate space could ever know. What this still cannot see, no experiment here can.

class pybnf.design.candidates.DesignExperiment(model: str, suffix: str, sim_data: Any, exp_data: Any, routing: Any)

One scored experiment at the best fit, with everything the design needs to reason about it.

sim_data is the simulated trajectory carrying the forward-sensitivity tensor, exp_data the measurements already taken, and routing the free-parameter-to-sensitivity-column map the gradient path built. model and suffix name the experiment, so a recommendation can say which one it is about; the suffix is also the scoring key the objective resolves a per-series scale against.

as_gradient_tuple()

The (sim_data, exp_data, routing, data_key) shape every gradient assembler takes.

property label

How this experiment is named in a report.

pybnf.design.candidates.baseline_information(objective, experiments, free_params)

The information the existing data already carries, at the best fit.

Every design is judged as an addition to this, because that is the question being asked: given what has already been measured, what should be measured next.

pybnf.design.candidates.candidate_information(objective, experiments, free_params, observables=None)

Enumerate every candidate measurement and the information each one would add.

Walks each experiment’s measured observables over every time on that experiment’s simulated grid. observables optionally restricts the walk to a named set of observables, for a user who can only run some of the assays.

Returns a CandidateSet. The order is stable – experiments in the order given, observables alphabetically, times in simulated order – so a design run is reproducible and ties break the same way every time.

pybnf.design.candidates.independent_variable(exp_data)

The name of an experiment’s independent variable: its first column, which is how every other consumer of a PyBNF dataset identifies it.

pybnf.design.candidates.measured_observables(objective, experiment)

The observables this experiment actually measures, in a stable order.

A column that is present but entirely blank is not measured, so it is left out: its noise model has never been exercised and there is no nearest real measurement to take a data-column noise scale from.

Criteria

Reducing an information matrix to one number, so two designs can be compared (#574).

The expected Fisher information F is a square matrix, one row and column per free parameter, in sampling space (ADR-0029 – so a log-scaled parameter’s entry is about its order of magnitude, which is the scale it is fitted on). Its inverse is the covariance matrix the fit would have, so (F^-1)_kk is the variance of parameter k and sqrt(threshold * (F^-1)_kk) is the half-width of that parameter’s confidence interval in the quadratic approximation – the same interval a profile-likelihood run traces, for the same threshold.

A criterion turns that matrix into a single score so designs can be ranked:

• 'a' – the summed variance of the parameters, or of a named subset. With one parameter named this is the classical c-criterion.

• 'd' – the log determinant, which is the volume of the joint confidence region.

• 'e' – the smallest eigenvalue, which is the worst-determined direction.

Singular information is not a numerical accident to be smoothed away. A parameter the data cannot constrain at all leaves a direction with no information in it, and the honest reading is an infinite variance, not a large one. Every function here decides that with one shared rule: an eigenvalue below SINGULAR_TOL times the largest one counts as zero, and a parameter with any weight on such a direction has infinite variance.

pybnf.design.criteria.CRITERIA = ('a', 'd', 'e')

The criteria a design run may be scored with.

pybnf.design.criteria.CRITERION_NAMES = {'a': 'A-optimal (average parameter variance)', 'd': 'D-optimal (confidence region volume)', 'e': 'E-optimal (worst-determined direction)'}

The full name of each criterion, for messages and report headers.

pybnf.design.criteria.NULL_COMPONENT_TOL = 1e-10

How much of a parameter’s own axis has to lie in the uninformed directions before its variance is infinite rather than merely large. Squared weights, so this is a very small angle.

pybnf.design.criteria.SINGULAR_TOL = 1e-10

An eigenvalue this far below the largest one carries no information. Relative, because the information matrix has the units of the data and can sit anywhere on the number line.

pybnf.design.criteria.criterion_score(information, criterion, targets=None)

The criterion as something to maximize, so the selection loop never has to ask which way a given criterion runs. The A-criterion’s summed variance is negated; the others are already amounts of information.

pybnf.design.criteria.criterion_value(information, criterion, targets=None)

The criterion read the way a person would state it: a summed variance for 'a', a log determinant for 'd', the smallest eigenvalue for 'e'.

targets is the list of parameter indices the A-criterion sums over (None -> all of them). Use lower_is_better() to know which direction is an improvement.

pybnf.design.criteria.interval_half_widths(information, threshold)

Each parameter’s confidence-interval half-width in sampling space, at the Delta chi2 threshold a profile-likelihood run would use.

This is the quadratic (Wald) approximation to the profile interval: the profile of a parameter near the optimum is the parabola Delta chi2 = (theta_k - theta*_k)^2 / (F^-1)_kk, which crosses the threshold at +- sqrt(threshold * (F^-1)_kk). For a linear model the approximation is exact. Infinite for a parameter with infinite variance, which reads as an open interval.

pybnf.design.criteria.is_singular(information)

Whether any direction in parameter space carries no information at all.

A singular information matrix means some combination of the parameters is invisible to the data. No amount of care with the criterion changes that; the design has to add a measurement that sees the missing direction.

pybnf.design.criteria.log_determinant(information)

log det F, or -inf when the information is singular (a confidence region of unbounded volume).

pybnf.design.criteria.lower_is_better(criterion)

Whether a smaller criterion_value() is the better design.

The A-criterion is a variance, so smaller is better; the other two are amounts of information, so larger is. criterion_score() hides this (it is always maximized) but a report has to say which way its numbers read.

pybnf.design.criteria.null_space_gain(information, block, targets=None)

How much of block falls in the directions information currently knows nothing about.

While the information is singular every criterion is pinned at its worst value – an infinite variance, a log determinant of -inf, a smallest eigenvalue of zero – so none of them can tell two candidates apart. This can, and it asks the right question at that moment: of the directions the data does not yet see, how much does this measurement see? The selection loop uses it until the information becomes invertible and then goes back to the requested criterion.

targets restricts the accounting to the uninformed directions that the target parameters actually lie along, so a c-criterion run is not sent off to fix a direction nobody asked about.

pybnf.design.criteria.parameter_variances(information)

Each parameter’s variance – the diagonal of the inverse information – in sampling space.

Infinite for a parameter that lies (even partly) along a direction the data does not constrain, which is the correct reading rather than a failure: no finite confidence interval exists for it. Computed from the eigendecomposition rather than by inverting, so the uninformed directions can be recognized instead of producing a huge finite number.

pybnf.design.criteria.smallest_eigenvalue(information)

The information in the worst-determined direction, floored at zero (a tiny negative eigenvalue is rounding: every term summed into the matrix is positive semi-definite).

pybnf.design.criteria.unidentified_parameters(information, param_names)

The parameters this information matrix leaves with an infinite variance.

Called on the information of the largest possible design – everything already measured plus every candidate at once – it names the parameters no experiment in the candidate space can pin down. That is a structural statement about the model and the observables, not a shortage of data, so it is reported rather than optimized around.

Selection

Choosing the best few measurements out of the whole candidate space (#574).

Picking the best five of two hundred candidate measurements is a subset-selection problem, not a continuous one: there are far too many subsets to try them all. The standard answer, and the one here, is to take the measurements one at a time, each time adding whichever remaining candidate improves the criterion most. It is not guaranteed to find the very best subset, but it is simple, it always terminates, and for the D-criterion it is the classical exchange algorithm’s forward half.

The same candidate may be chosen more than once. That is not a bug to be suppressed: choosing a point twice means measuring it twice, and it is exactly the right recommendation when the precision of one measurement, rather than the shape of the trajectory, is what limits you.

One special case has to be handled explicitly. When the information matrix says nothing at all about a direction the criterion cares about, the criterion sits at its worst possible value for every candidate and cannot tell them apart – an infinite variance is an infinite variance whatever you add to it. The selection loop notices and asks a different question until that stops being true: of the directions nothing yet sees, which candidate sees the most (null_space_gain())? Then it goes back to the requested criterion for the rest of the picks. A design aimed at one parameter is not blocked by some other combination of parameters being invisible, so this only fires when the target itself is the problem.

class pybnf.design.greedy.DesignResult(criterion: str, targets: list, target_names: list, measurements: list = <factory>, trace: list = <factory>, baseline: ~typing.Any = None, information: ~typing.Any = None, escaped_singular: int = 0, truncated: bool = False)

A finished design: what to measure, and what it is expected to buy.

measurements are in the order they were chosen, repeats included, so reading them in order shows how much each successive measurement is still worth. baseline is the information the existing data already carries and information is what it would become; trace is the criterion after each pick, so a design that stops paying off is visible rather than implied.

property baseline_value

The criterion before any of the recommended measurements are made.

grouped()

The recommendation as (measurement, replicates, first_rank) in the order the measurements were first chosen – the reading a person wants, where choosing one point three times is one row saying “measure it three times”.

property value

The criterion once every recommended measurement has been made.

pybnf.design.greedy.improvement(result)

How much better the criterion got, as a plain ratio, or None when it cannot be stated.

For the A-criterion this is the factor the summed variance shrank by, so 4.0 means the variance is a quarter of what it was and the confidence interval is half as wide. A baseline that was infinite (a parameter the existing data cannot determine at all) has no ratio, which is itself the headline: the design goes from no answer to an answer.

pybnf.design.greedy.require_identifiable(baseline, candidates, param_names, targets)

Refuse a design whose targets no experiment in the candidate space could ever pin down.

Adding every candidate at once is the most a design over this space can know. A target still left with an infinite variance there is not short of data: no measurement of these observables, at any simulated time, tells the model apart along that direction. That is structural non-identifiability, and the fix is a different model or a different observable, not a different design.

pybnf.design.greedy.resolve_targets(variables, names, criterion)

The parameter indices the A-criterion sums variances over, validated against the fit.

names empty or absent means every free parameter. Naming exactly one makes this the classical c-criterion: minimize the variance of that one parameter, which is what a profile-likelihood verdict about a single parameter asks for.

Targets are refused for the D and E criteria rather than quietly ignored. Both are properties of the whole information matrix – a volume and a worst direction – so restricting them to a subset of parameters means something different from what a reader would assume, and a request that cannot be honoured as written should say so.

pybnf.design.greedy.select_design(baseline, candidates, n_points, criterion, targets, param_names)

Choose n_points measurements, one at a time, each the best next addition.

baseline is the information the existing data carries, candidates the CandidateSet to choose from. Returns a DesignResult. Ties break toward the earliest candidate, which is the earliest time of the first observable of the first experiment, so the same inputs always give the same design.

Report

Writing a design down: what to measure, and what it is expected to buy (#574).

The report has two halves, because a recommendation nobody can check is not worth much.

The first half is the recommendation itself: the measurements to make, in the order they were chosen, with a count when the same point is chosen more than once. The second half is the reason, stated in the units the user has already seen from a profile-likelihood run – each parameter’s confidence interval as it stands now, and as it would be once the recommended measurements are in hand. Those predicted intervals come from the same information matrix the design was chosen with, read through the quadratic approximation to the profile: the interval is theta* +- sqrt(threshold * variance) in the parameter’s own fitted scale. For a linear model that is exact, and for anything else it is the local approximation, which is also all the design itself ever claimed to be.

pybnf.design.report.format_design_summary(result, variables, u_star, threshold)

The same design as a short block of lines for the terminal: what to measure, and what it does to the intervals of the parameters the design was aimed at.

pybnf.design.report.predicted_intervals(result, variables, u_star, threshold)

Every parameter’s interval before and after the design, plus how much it shrinks.

Each row is a dict with the parameter’s name, its value at the best fit, the interval the existing data supports, the interval the design would support, and width_ratio – the designed half-width over the current one, so 0.5 means the interval halves. The ratio is None when the current interval is open, which is the strongest result there is: the design replaces no answer with an answer.

pybnf.design.report.write_design_report(path, result, variables, u_star, threshold, confidence)

Write the design report to path as a tab-delimited file with commented headers, the same shape as the profile-likelihood summary beside it.