Follow-up to #574, which shipped the local half in #668.
job_type = design computes the expected Fisher information at the best fit and recommends measurements from it. That means the design is only as good as the fit it was computed at. If the point estimate is off, a design optimized there can be worthless, and the run has no way to say so.
The usual answer is to average the criterion over an ensemble of plausible parameter values instead of evaluating it at one point. PyBNF already produces two suitable ensembles for free:
- posterior draws from
am, dream, or p_dream;
- the points a
profile_likelihood run already computed and wrote to Results/profile_<name>.txt, each of which carries the full re-optimized parameter set.
The second is the cheap one, and reusing profile points this way is not something other tools appear to do.
What it would take
The pieces that choose and score a design do not need to change. pybnf.design already separates the information matrix from the criterion and from the selection, so a robust version is a different way of producing the information: one matrix per ensemble member, averaged, or the criterion averaged over the members, which are not the same thing and the choice needs deciding.
The cost is the reason this was deferred. Each ensemble member needs its own simulation with sensitivities, so an ensemble of a hundred points is a hundred evaluations. They are independent, so they would farm across the scheduler the way profile-likelihood tracks already do.
What has to be shown before shipping it
The evaluation #574 asked for and this issue inherits: quantify the gap between the local design and the robust design over the profile ensemble on a real model. If the local design turns out to be stable across the ensemble, the robust version is not worth its cost and should not ship. If it is unstable, that instability is itself worth reporting to the user, and the local design should probably say when it is on shaky ground.
Open questions
- Average the criterion over members, or average the information and then take the criterion? The first is the standard Bayesian design criterion. The second is cheaper and is not the same answer.
- How should ensemble members be weighted? Posterior draws are already distributed correctly. Profile points are not a sample from anything, so using them as one needs a stated justification, or a weighting by their likelihood.
- Should the ensemble be thinned? A hundred simulations to place five measurements may not be a trade a user wants without being asked.
Follow-up to #574, which shipped the local half in #668.
job_type = designcomputes the expected Fisher information at the best fit and recommends measurements from it. That means the design is only as good as the fit it was computed at. If the point estimate is off, a design optimized there can be worthless, and the run has no way to say so.The usual answer is to average the criterion over an ensemble of plausible parameter values instead of evaluating it at one point. PyBNF already produces two suitable ensembles for free:
am,dream, orp_dream;profile_likelihoodrun already computed and wrote toResults/profile_<name>.txt, each of which carries the full re-optimized parameter set.The second is the cheap one, and reusing profile points this way is not something other tools appear to do.
What it would take
The pieces that choose and score a design do not need to change.
pybnf.designalready separates the information matrix from the criterion and from the selection, so a robust version is a different way of producing the information: one matrix per ensemble member, averaged, or the criterion averaged over the members, which are not the same thing and the choice needs deciding.The cost is the reason this was deferred. Each ensemble member needs its own simulation with sensitivities, so an ensemble of a hundred points is a hundred evaluations. They are independent, so they would farm across the scheduler the way profile-likelihood tracks already do.
What has to be shown before shipping it
The evaluation #574 asked for and this issue inherits: quantify the gap between the local design and the robust design over the profile ensemble on a real model. If the local design turns out to be stable across the ensemble, the robust version is not worth its cost and should not ship. If it is unstable, that instability is itself worth reporting to the user, and the local design should probably say when it is on shaky ground.
Open questions