Skip to content

LOO refactoring and adding pred_measure feature#363

Draft
florence-bockting wants to merge 5 commits into
masterfrom
pred_measure
Draft

LOO refactoring and adding pred_measure feature#363
florence-bockting wants to merge 5 commits into
masterfrom
pred_measure

Conversation

@florence-bockting
Copy link
Copy Markdown
Contributor

@florence-bockting florence-bockting commented Jun 2, 2026

Fixes #281

This PR is work in progress and the description will be refined over time.

Description

This PR is a larger project with the aim to refactor the loo package and to add new features related to a new pred_measure family.

In the course of refactoring this PR aims to address also the following PRs:

How to navigate this PR

As this is a large, long-running PR it might help to first read through the supporting documents that explain the design, scope, and current status:

  • developer-notes.md: The central reference for this project: current status, open tasks, outstanding design decisions, and known blockers. Start here.
  • pred_measure-formulas.Rmd: An overview of all supported scores and metrics, including their mathematical definitions.
  • pred_measure-family.Rmd: A conceptual introduction to the new pred_measure family and its intended usage.

Acknowledgements

This work builds on an initial prototype of the new pred_measure feature implemented by @VisruthSK under guidance from @jgabry (see VisruthSK/loo-sandbox).

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Jun 2, 2026

This is how benchmark results would change (along with a 95% confidence interval in relative change) if 7649d9c is merged into master:

  • ✔️loo_function: 1.75s -> 1.75s [-0.29%, +0.62%]
  • ✔️loo_matrix: 1.32s -> 1.32s [-0.7%, +0.4%]
    Further explanation regarding interpretation and methodology can be found in the documentation.

@florence-bockting
Copy link
Copy Markdown
Contributor Author

florence-bockting commented Jun 8, 2026

Linking here to Issue #179 as it seems to be about loo scores. We should see whether it is still relevant and can be implemented within loo refactoring.

@florence-bockting
Copy link
Copy Markdown
Contributor Author

florence-bockting commented Jun 8, 2026

There is an open issue requesting a subset() method for psis objects #343 and an (unrelated) open PR #110. Might be worth considering whether we can address this as well in this refactoring PR. If yes, we should

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment