Recommend the measurements to make next (#574) - #668
Merged
Conversation
A profile-likelihood run ends by telling you a parameter is practically non-identifiable, which is a diagnosis with no prescription. This adds the prescription. The expected Fisher information PyBNF already assembles for the gntr optimizer is a plain sum over the measured points, so the information a planned measurement would add is that measurement's own term. A new generator yields those terms one at a time instead of summed, and a design scores a candidate measurement by handing the existing assembly a one-row dataset holding the model's own prediction there. Every noise model, scale and transform the fit supports comes along, and no model is re-simulated to enumerate or score the candidates. Two surfaces. job_type = design takes the fitted values and writes a report. profile_likelihood_design = 1 makes an identifiability run end with the same report, aimed at the parameters it just flagged. Both off or absent by default. The report is quoted in the units a profile-likelihood run already uses: each parameter's confidence interval now and after. A recovery test checks that claim by making the recommended measurements and profiling again. The two agree to about a tenth of a percent on the decay model. For a time course PyBNF simulates the times the data was measured at, so by default a design could only recommend repeating a measurement. design_grid and design_t_end add extra simulated times to choose from, out past the last measurement. The measured times are kept, so scoring is unchanged. Design recorded in ADR-0129.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #574.
A profile-likelihood run ends by telling you that a parameter is practically non-identifiable. That is a diagnosis with no prescription, and the next question is always "then what experiment should I run?". This adds the answer.
How it works
The expected Fisher information PyBNF already assembles for the
gntroptimizer is a plain sum over the measured points, one small matrix per measurement. So the information a planned measurement would add is just that measurement's own term. A new public generator,iter_fisher_points, yields those terms one at a time instead of summed, and a candidate measurement is scored by handing the existing Fisher assembly a one-row dataset holding the model's own prediction at that point. That is what expected Fisher information means, and it is why every noise model, scale, transform and constraint the fit already supports comes along without a special case. Summing everything the generator yields reproducesassemble_fisher_hessianexactly, and that shipped path is untouched.No model is re-simulated to enumerate or score candidates. The sensitivities at every simulated time were computed when the best fit was scored.
Two ways to get a design
job_type = designtakes the fitted values from aninitial_value:on every parameter, exactly asprofile_likelihoodtakes its optimum, simulates that one point, and writes the report.profile_likelihood_design = 1makes an identifiability run end by writing the same report, around the optimum it just found and aimed at the parameters it just flagged as practically non-identifiable. Both are off or absent by default, so no existing run changes.What it recommends
One observable, in one experiment, at one time. The observable has to be one that experiment already measures, so its precision is the one your fit is already using rather than a number invented for an assay nobody has run. The same measurement can be recommended twice, which means measure it twice.
design_criterionsays what makes one design better:afor the average parameter variance, aimed at the parameters named bydesign_target(the classical c-criterion when one is named, which is what a verdict about a single parameter asks for),dfor the volume of the joint confidence region,efor the worst-determined direction.If measuring every candidate at once still leaves a target parameter undetermined, the run says so instead of recommending measurements that cannot help. That is structural non-identifiability, and the fix is a different observable or a different model.
One thing worth knowing before you use it
For a time course PyBNF derives the simulated grid from the data. So with nothing else in place a design could only ever recommend repeating a measurement you have already made.
design_grid = Nadds N extra simulated times to choose from anddesign_t_endmoves the far end of that window past your last measurement. Your measured times are always kept in the grid, so the scoring of your data is unchanged. This follows the precedent ADR-0112 set fortime_error, which also needed a data-bearing time course simulated on a grid the data did not dictate.Does the recommendation actually work
The report is quoted in the units a profile-likelihood run already uses: each parameter's confidence interval now, and after the recommended measurements are in hand.
A recovery test checks that claim by making the measurements. It profiles
kagainst a short early window of decay data, runs a design aimed atk, generates the recommended measurements from the model, and profiles again with them added. The interval the second run traces is 0.172 times as wide as the first. The design predicted 0.172.The offline tests check the answers against results that are known in advance: the D-optimal design for a straight line on an interval measures its two ends, the most informative time for a decay rate is one lifetime, and two parameters that only ever appear added together cannot be separated by any measurement. There is also a consistency check against
assemble_fisher_hessianassembled on a dataset that literally contains the recommended measurements, so the planned-measurement construction is not merely self-consistent.Not in this pull request
Robust design, which averages the criterion over an ensemble of plausible parameter values rather than computing it at one point, is filed separately as #669. Proposing a new experimental condition is out of scope, because a perturbation is a model modification and PyBNF has no vocabulary for proposing one.
Testing
Full local gate green, 4625 passed. Design tests: 33 offline plus 4 in the opt-in recovery tier. Docs build clean. Design recorded in ADR-0129.