1185: Support incremental update of materialized samples view#7726
Open
labkey-nicka wants to merge 10 commits into
Open
1185: Support incremental update of materialized samples view#7726labkey-nicka wants to merge 10 commits into
labkey-nicka wants to merge 10 commits into
Conversation
XingY
reviewed
Jun 5, 2026
labkey-nicka
force-pushed
the
fb_incremental_update
branch
from
9e3ba8b to
219f73f
Compare
labkey-nicka
commented
Jun 5, 2026
| { | ||
| SqlDialect d = CoreSchema.getInstance().getSchema().getSqlDialect(); | ||
| // SQL Server's datetime type lacks microsecond precision | ||
| Timestamp comparison = d.isSqlServer() ? new Timestamp(changedSince.getTime() - 500) : changedSince; |
Contributor
There was a problem hiding this comment.
2ms should do it (not that it matters)
Contributor
Author
There was a problem hiding this comment.
Noted. Left as-is.
| .append("\nUNION ALL\n") | ||
| .append(getViewSourceSql() | ||
| .append(" AND m.rootmaterialrowid <> m.rowid") | ||
| .append(" AND EXISTS (SELECT 1 FROM exp.material r WHERE r.rowid = m.rootmaterialrowid AND r.cpastype = ").appendValue(_lsid, d) |
Contributor
There was a problem hiding this comment.
Just thiking out loud: If your new index is (cpastype, modified) INCLUDE (rowid) Then (SELECT rowid FROM exp.material WHERE cpasType=? and modified >=?) would be extremely fast. This could be maybe simplified to
WITH modified AS (SELECT rowid FROM...)
getSelectSQL() WHERE rowid in (SELECT rowid FROM modified)
UNION
getSqlectSQL() WHERE rootmaterialrowid IN (SELECT rowid FROM modified)
(deduped with UNION or with rowid=/<>rootmaterialrowid checks as in your original query)
I'm not sure it's faster, but maybe?
Contributor
Author
There was a problem hiding this comment.
I agree this is more straightforward. Query updated to use UNION. I did not modify the index as rowId is already indexed so table scanning will not be required.
labkey-nicka
force-pushed
the
fb_incremental_update
branch
from
aaf4b5b to
51d9e28
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale
This seeks to address a portion of #1185 by introducing support for incremental update of the materialized view. The rows to update are determined by checking the modified column against a timestamp created when the mutating action is initiated. Additionally, supports incremental insert + update when it is a merge action.
Related Pull Requests
Changes
ExpMaterialTableImplchangedSincetimestamp parameterTasks
Manual testing note
Fail: Move samples triggers full table materialize