Skip to content

Keep rank-deficient lm and glm fits predictable - #369

Open
EmilHvitfeldt wants to merge 1 commit into
mainfrom
rank-deficient-lm
Open

Keep rank-deficient lm and glm fits predictable#369
EmilHvitfeldt wants to merge 1 commit into
mainfrom
rank-deficient-lm

Conversation

@EmilHvitfeldt

Copy link
Copy Markdown
Member

Ticks only the first box of #308. Items 2 (label parsing) and 3 (prediction intervals and case weights) are untouched and stay open.

The problem

parse_model_lm() called qr.solve(qr.R(model$qr)) on every parse. A rank-deficient fit has a singular R factor, so the parse aborted for a model whose fitted values are well defined, even though tidypredict_fit() never touches the QR decomposition.

The issue understates the reach: two everyday shapes fail, not one.

set.seed(1); n <- 120
d <- data.frame(x1 = rnorm(n), x2 = runif(n, 0, 10))
d$y <- 2 * d$x1 - 0.5 * d$x2 + rnorm(n, sd = 0.3)

tidypredict_fit(lm(y ~ x1 + xdup, transform(d, xdup = d$x1)))    # duplicated column
tidypredict_fit(lm(y ~ x1 + xconst, transform(d, xconst = 1)))   # zero-variance predictor
#> Error: Unable to calculate inverse of QR decomposition.

A predictor with no variance is not mentioned in the issue and is the more realistic way to hit this. Both fail identically for glm(), gaussian and binomial. predict() handles all of them.

The fix

Drop the NA coefficients R leaves for the terms it could not identify, as predict() does, and build the inverse from the leading rank rows and columns of the R factor: exactly the columns whose coefficients survive. That block is not singular, so prediction intervals keep working for rank-deficient models as well, which was better than expected.

The QR failure no longer aborts the parse. qr_inverse_lm() returns NULL when no inverse can be formed and te_interval_lm() reports it with a cli message, which is the only place the QR is required. Previously that path failed with Must supply `.init` when `.x` is empty. This also removes the grepl("singular matrix", cnd$message) test on an untranslated English message and the bare stop(cnd) re-raise, both noted as secondary in the issue.

Measured

Max absolute difference against the modelling package's own predict(), 120 rows:

model shape fit interval
lm duplicated column 8.9e-16 8.9e-16
lm zero-variance predictor 8.9e-16 8.9e-16
lm mpg ~ cyl_f * gear_f, 1 aliased coef 0 n/a
glm gaussian duplicated column 8.9e-16 n/a
glm binomial zero-variance predictor 1.1e-16 n/a
lm full rank (regression check) mpg ~ wt + disp 3.6e-15 0

Intervals compared against predict(interval = "prediction")[, "upr"].

Other classes checked

parse_model_lm() is shared by lm, glm and rq. quantreg::rq() refuses to fit a singular design itself and stores no $qr, so it takes the same NULL path as before: unchanged. The parsnip linear_reg() lm and glm engines route through the same parser and now work on rank-deficient fits (4.4e-16 against predict()). The other callers of build_terms() (lda, qda, fda, sda, sparsediscrim, mixOmics, multinom, glmnet) pass qr = NULL and were not touched.

Tests

New tests in tests/testthat/test-model-lm.R for both shapes, for lm and glm, compared against stats::predict(). Each was confirmed to fail before the fix (4 failures on the reverted source). The old #124 snapshot test asserted the abort that this PR removes, so it now asserts the fitted values match predict() instead.

devtools::test(): FAIL 0 | WARN 13 | SKIP 0 | PASS 2002 (the warnings are the stale local H2O cluster). devtools::check() is clean apart from a hidden-directory NOTE from the worktree and one H2O REST failure in test-h2o.R:142, both unrelated to this change.

`parse_model_lm()` inverted the full R factor of the model's QR
decomposition on every parse, even though `tidypredict_fit()` never
uses it. A rank-deficient fit has a singular R factor, so the whole
parse aborted with "Unable to calculate inverse of QR decomposition"
for a model whose fitted values are perfectly well defined.

Two everyday shapes hit this, not just the exactly-collinear one the
issue mentions: a duplicated predictor column, and a predictor with no
variance at all. Both fail identically for `lm()` and for `glm()`.

R leaves the coefficients it could not identify as `NA` and `predict()`
drops those terms. Do the same, and build the inverse from the leading
`rank` rows and columns of R, which are exactly the columns whose
coefficients survive. That block is not singular, so prediction
intervals keep working for these models too, matching
`predict(interval = "prediction")` to 9e-16.

The QR failure no longer aborts the parse at all: the inverse is `NULL`
when it cannot be formed, and `te_interval_lm()` reports it, which is
the only place the QR is needed. That also removes the
`grepl("singular matrix", ...)` test on an untranslated English message
and the bare `stop(cnd)` re-raise.

Fixes the first item of #308.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant