Skip to content

Recommend the measurements to make next (#574) - #668

Merged
wshlavacek merged 1 commit into
mainfrom
oed-design-574
Aug 25, 2026
Merged

Recommend the measurements to make next (#574)#668
wshlavacek merged 1 commit into
mainfrom
oed-design-574

Conversation

@wshlavacek

@wshlavacek wshlavacek commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Closes #574.

A profile-likelihood run ends by telling you that a parameter is practically non-identifiable. That is a diagnosis with no prescription, and the next question is always "then what experiment should I run?". This adds the answer.

How it works

The expected Fisher information PyBNF already assembles for the gntr optimizer is a plain sum over the measured points, one small matrix per measurement. So the information a planned measurement would add is just that measurement's own term. A new public generator, iter_fisher_points, yields those terms one at a time instead of summed, and a candidate measurement is scored by handing the existing Fisher assembly a one-row dataset holding the model's own prediction at that point. That is what expected Fisher information means, and it is why every noise model, scale, transform and constraint the fit already supports comes along without a special case. Summing everything the generator yields reproduces assemble_fisher_hessian exactly, and that shipped path is untouched.

No model is re-simulated to enumerate or score candidates. The sensitivities at every simulated time were computed when the best fit was scored.

Two ways to get a design

job_type = design takes the fitted values from an initial_value: on every parameter, exactly as profile_likelihood takes its optimum, simulates that one point, and writes the report.

profile_likelihood_design = 1 makes an identifiability run end by writing the same report, around the optimum it just found and aimed at the parameters it just flagged as practically non-identifiable. Both are off or absent by default, so no existing run changes.

What it recommends

One observable, in one experiment, at one time. The observable has to be one that experiment already measures, so its precision is the one your fit is already using rather than a number invented for an assay nobody has run. The same measurement can be recommended twice, which means measure it twice.

design_criterion says what makes one design better: a for the average parameter variance, aimed at the parameters named by design_target (the classical c-criterion when one is named, which is what a verdict about a single parameter asks for), d for the volume of the joint confidence region, e for the worst-determined direction.

If measuring every candidate at once still leaves a target parameter undetermined, the run says so instead of recommending measurements that cannot help. That is structural non-identifiability, and the fix is a different observable or a different model.

One thing worth knowing before you use it

For a time course PyBNF derives the simulated grid from the data. So with nothing else in place a design could only ever recommend repeating a measurement you have already made. design_grid = N adds N extra simulated times to choose from and design_t_end moves the far end of that window past your last measurement. Your measured times are always kept in the grid, so the scoring of your data is unchanged. This follows the precedent ADR-0112 set for time_error, which also needed a data-bearing time course simulated on a grid the data did not dictate.

Does the recommendation actually work

The report is quoted in the units a profile-likelihood run already uses: each parameter's confidence interval now, and after the recommended measurements are in hand.

A recovery test checks that claim by making the measurements. It profiles k against a short early window of decay data, runs a design aimed at k, generates the recommended measurements from the model, and profiles again with them added. The interval the second run traces is 0.172 times as wide as the first. The design predicted 0.172.

The offline tests check the answers against results that are known in advance: the D-optimal design for a straight line on an interval measures its two ends, the most informative time for a decay rate is one lifetime, and two parameters that only ever appear added together cannot be separated by any measurement. There is also a consistency check against assemble_fisher_hessian assembled on a dataset that literally contains the recommended measurements, so the planned-measurement construction is not merely self-consistent.

Not in this pull request

Robust design, which averages the criterion over an ensemble of plausible parameter values rather than computing it at one point, is filed separately as #669. Proposing a new experimental condition is out of scope, because a perturbation is a model modification and PyBNF has no vocabulary for proposing one.

Testing

Full local gate green, 4625 passed. Design tests: 33 offline plus 4 in the opt-in recovery tier. Docs build clean. Design recorded in ADR-0129.

A profile-likelihood run ends by telling you a parameter is practically
non-identifiable, which is a diagnosis with no prescription. This adds the
prescription.

The expected Fisher information PyBNF already assembles for the gntr optimizer is
a plain sum over the measured points, so the information a planned measurement
would add is that measurement's own term. A new generator yields those terms one
at a time instead of summed, and a design scores a candidate measurement by
handing the existing assembly a one-row dataset holding the model's own
prediction there. Every noise model, scale and transform the fit supports comes
along, and no model is re-simulated to enumerate or score the candidates.

Two surfaces. job_type = design takes the fitted values and writes a report.
profile_likelihood_design = 1 makes an identifiability run end with the same
report, aimed at the parameters it just flagged. Both off or absent by default.

The report is quoted in the units a profile-likelihood run already uses: each
parameter's confidence interval now and after. A recovery test checks that claim
by making the recommended measurements and profiling again. The two agree to
about a tenth of a percent on the decay model.

For a time course PyBNF simulates the times the data was measured at, so by
default a design could only recommend repeating a measurement. design_grid and
design_t_end add extra simulated times to choose from, out past the last
measurement. The measured times are kept, so scoring is unchanged.

Design recorded in ADR-0129.
@wshlavacek
wshlavacek merged commit 6cbb5a2 into main Aug 25, 2026
9 checks passed
@wshlavacek
wshlavacek deleted the oed-design-574 branch August 25, 2026 11:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant