Keep rank-deficient lm and glm fits predictable - #369
Open
EmilHvitfeldt wants to merge 1 commit into
Open
Conversation
`parse_model_lm()` inverted the full R factor of the model's QR
decomposition on every parse, even though `tidypredict_fit()` never
uses it. A rank-deficient fit has a singular R factor, so the whole
parse aborted with "Unable to calculate inverse of QR decomposition"
for a model whose fitted values are perfectly well defined.
Two everyday shapes hit this, not just the exactly-collinear one the
issue mentions: a duplicated predictor column, and a predictor with no
variance at all. Both fail identically for `lm()` and for `glm()`.
R leaves the coefficients it could not identify as `NA` and `predict()`
drops those terms. Do the same, and build the inverse from the leading
`rank` rows and columns of R, which are exactly the columns whose
coefficients survive. That block is not singular, so prediction
intervals keep working for these models too, matching
`predict(interval = "prediction")` to 9e-16.
The QR failure no longer aborts the parse at all: the inverse is `NULL`
when it cannot be formed, and `te_interval_lm()` reports it, which is
the only place the QR is needed. That also removes the
`grepl("singular matrix", ...)` test on an untranslated English message
and the bare `stop(cnd)` re-raise.
Fixes the first item of #308.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ticks only the first box of #308. Items 2 (label parsing) and 3 (prediction intervals and case weights) are untouched and stay open.
The problem
parse_model_lm()calledqr.solve(qr.R(model$qr))on every parse. A rank-deficient fit has a singular R factor, so the parse aborted for a model whose fitted values are well defined, even thoughtidypredict_fit()never touches the QR decomposition.The issue understates the reach: two everyday shapes fail, not one.
A predictor with no variance is not mentioned in the issue and is the more realistic way to hit this. Both fail identically for
glm(), gaussian and binomial.predict()handles all of them.The fix
Drop the
NAcoefficients R leaves for the terms it could not identify, aspredict()does, and build the inverse from the leadingrankrows and columns of the R factor: exactly the columns whose coefficients survive. That block is not singular, so prediction intervals keep working for rank-deficient models as well, which was better than expected.The QR failure no longer aborts the parse.
qr_inverse_lm()returnsNULLwhen no inverse can be formed andte_interval_lm()reports it with a cli message, which is the only place the QR is required. Previously that path failed withMust supply `.init` when `.x` is empty. This also removes thegrepl("singular matrix", cnd$message)test on an untranslated English message and the barestop(cnd)re-raise, both noted as secondary in the issue.Measured
Max absolute difference against the modelling package's own
predict(), 120 rows:lmlmlmmpg ~ cyl_f * gear_f, 1 aliased coefglmgaussianglmbinomiallmfull rank (regression check)mpg ~ wt + dispIntervals compared against
predict(interval = "prediction")[, "upr"].Other classes checked
parse_model_lm()is shared bylm,glmandrq.quantreg::rq()refuses to fit a singular design itself and stores no$qr, so it takes the sameNULLpath as before: unchanged. The parsniplinear_reg()lmandglmengines route through the same parser and now work on rank-deficient fits (4.4e-16 againstpredict()). The other callers ofbuild_terms()(lda,qda,fda,sda,sparsediscrim,mixOmics,multinom,glmnet) passqr = NULLand were not touched.Tests
New tests in
tests/testthat/test-model-lm.Rfor both shapes, forlmandglm, compared againststats::predict(). Each was confirmed to fail before the fix (4 failures on the reverted source). The old#124snapshot test asserted the abort that this PR removes, so it now asserts the fitted values matchpredict()instead.devtools::test():FAIL 0 | WARN 13 | SKIP 0 | PASS 2002(the warnings are the stale local H2O cluster).devtools::check()is clean apart from a hidden-directory NOTE from the worktree and one H2O REST failure intest-h2o.R:142, both unrelated to this change.