LMDB/SQLite store, temporal APIs, shared-API parity, SQLite Share_index_db - #5
Open
tiensonqin wants to merge 90 commits into
Open
LMDB/SQLite store, temporal APIs, shared-API parity, SQLite Share_index_db#5tiensonqin wants to merge 90 commits into
tiensonqin wants to merge 90 commits into
Conversation
Define a repository-managed environment for datascript-ocaml: - Base image installs opam, OCaml 5.5.0, libsqlite3-dev, and Node.js 24 - Install script runs opam deps and dune build after checkout Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Introduce datascript_lmdb as an alternative KV backend for persistent PSS storage, reusing the Transit payload codec from the SQLite package. Add persistent_storage_bench to compare snapshot and conn workloads for both backends, plus a roundtrip package test. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Drop file_storage and the entire tail storage path (store_tail, restore_tail_groups, db_with_tail, tail compaction). Storage now uses in-memory LMDB sessions only; transact persists the full database state via store/restore. Update public APIs, platform storage modules, and tests accordingly. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- Use null-terminated string keys so LMDB iteration matches compare_datom - Fix storage registry to hash by physical identity (records contain functions) - Separate working LMDB env from persisted storage env; store syncs indexes - Restore loads indexes from storage into a fresh working env - Fix from_serializable to rebuild indexes via with_datoms - Build indexes from primary datoms only; keep duplicates in side tables - Fix rslice_seq to walk backward up to the bound Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Use functional overlay indexes so transact returns new db handles without mutating the input db. Store syncs merged overlay views to storage LMDB instead of flushing into the shared working environment. Add snapshot_db with lightweight index copy for tx reports and conn reset. Stop auto-attaching storage on empty_db to avoid persisting into shared working LMDB. Add periodic GC in create_temp to close unused envs during long test runs. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- Add with_write_txn, put/remove/copy_index_txn, and fold_index_range on LMDB db - Batch of_sorted_list, flush, sync_merged_to_lmdb, and storage sync in single txns - Use cursor seek for slice lower bounds; keep custom cmp filtering for exact prefixes - Fix sync_merged_to_lmdb to write into the target env (not the working env txn) - Add 20k PSS vs LMDB benchmark harness for regression tracking Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- Load indexes via Index.of_bulk at init instead of writing 240k LMDB keys upfront - Keep sorted attr arrays and (attr,value) entity-id index for AVET lookups - Add cursor/bulk fast paths in LMDB index fold, slice, find, and sync - Route constant query patterns through datoms_by_attr_value in query_where - Cache Marshal-encoded datom payloads during bulk writes Benchmarks (20k entities, vs PSS): build-all-init 0.73x, scan-aevt-name 0.67x, storage-roundtrip 0.59x. query-name-ivan and add-one-tx still slower than PSS. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- Use sorted bulk arrays for O(log n) range slices instead of scanning 80k overlays - Keep O(1) bulk Index.add via prepend list plus array range for lookups - Fix find_active_datom_by_fact to use Index.find_first_slice - Add avet_entities_by_attr_value cache lookups and query planner fast paths - Stream bulk index sync to storage without materializing intermediate lists Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- Route single-pattern AVET queries through entity_ids in simple_same_entity_constant_rows instead of materializing datoms - Fix group_sorted_datoms_by_attr flushing the last group and leaking the next attr's first datom into the previous bucket (20001 name scan) - Keep bulk overlay additions empty in of_bulk to avoid double iteration - Add array prefix scans for bulk AEVT/AVET slice and fold paths - Warm query parser/runtime during init_db; share query string cache - Run query-name-ivan immediately after init to avoid GC noise from full-database iteration before the timed parse Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Introduce tx visibility filtering and public history/time-travel API matching dbval.core: basis_tx, as_of/as_of_t, since/since_t, history, temporal_view. - db fields: max_tx (basis), store_max_tx, as_of_tx, since_tx, history - tx_visibility module with datoms_filter matching dbval semantics - transact rejects temporal views with dbval-compatible error message - design doc for overlay removal and append-only migration Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Replace overlay merge model with append-only LMDB writes and dbval-style
tx visibility on all read paths.
- Simplify Index.t to { db, which }; remove additions/removals/bulk
- append_tx_data: single LMDB txn for EAVT/AEVT/AVET on transact
- init via of_eavt_datoms (one txn); snapshot_db is O(1) shared handle
- refresh_indexes_with_tx_data appends full tx_data (add + retract)
- apply_db_view on datoms/eavt/attr caches for datoms-filter + basis
- test_tx_history: as_of, since, history integration tests
Known regression: add-one-tx ~1.4ms vs ~0.01ms overlay (LMDB write cost).
Store still copies full index when session/storage envs differ.
Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Defer LMDB writes for incremental transacts on databases without attached storage into db.pending_datoms. Bulk init still writes session LMDB directly. Store flushes pending via flush_pending_datoms before syncing indexes. Read paths merge pending_overlay with LMDB cursors without forcing full list materialization when only duplicates are absent. Fix find_eavt/find_avet and exact-prefix/seek paths to include pending datoms. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Add sync_append_since_tx to copy only datoms with tx > stored meta max_tx when session and storage LMDB envs differ. Skip index copy when envs are shared (storage-attached dbs). Add test_storage multi-tx incremental store with as_of/history after restore. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Implement :db/purge, :db.purge/attribute, and :db.purge/entity transaction operations that physically remove datoms from current and history views, matching Datahike purge semantics. Purge searches the history stream, deletes keys from append-only LMDB indexes, and syncs removals to persistent storage. Also includes tx-filter history fixes (codec added flag, transact read ceiling, public temporal API) and test_purge regression coverage. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Cover basis_tx, as_of/since bounds, history retractions, entity retraction trails, temporal view transact guards, view immutability, index parity, and public API aliases. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- Replace fixed storage_kind variants with extensible string labels (storage_kind_memory/lmdb/sqlite constants for built-in names) - Unify all backends behind storage_backend callbacks registered via Datascript_storage_protocol.register_backend - Default datascript-ocaml-native.storage package: memory only - Optional opam packages: datascript-ocaml-native-lmdb and datascript-ocaml-native-sqlite with plugin modules - Migrate storage/history/purge tests to Alcotest Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Since tx is exclusive (tx > since_tx), post-bootstrap age updates appear in since tx0 history, not since tx1 where age 30 was asserted. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- Delete dead sync_merged_to_lmdb and stale lmdb root copies; storage lives under lmdb/native and storage/native only. - Route fold_datoms through datoms() on temporal views so tx-filter/history apply to full-index scans. - Replace melange LMDB db with in-memory Hashtbl backend matching native API; enable byte mode for jsoo via melange lmdb/storage libraries. - Add sqlite/datascript_sqlite_db.ml and tidy codec imports. - Update design doc and bench row_count after list_addresses removal. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- bench/datahike_compare.ml: shared-db query suite aligned with Datahike datascript-bench (15 queries, same timing protocol). - bench/compare_ocaml_datahike.sh + datahike_shared_bench.clj for side-by-side runs. - query_where: route [?e :attr ?v] + comparison predicates through avet index_range instead of rejecting with constant_patterns=[] (fixes qpred1/2/range scan path). - test/test_datahike_queries.ml: golden result-count parity tests for key queries. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- Add docs/adr/query-planner.md and docs/query_planner_plan.md for compiled planner direction without new public APIs. - Route constant+value-var same-entity queries through the relation evaluator instead of the incomplete simple_same_entity fast path (fixes q2/q2-switch). - Fall back to hash-join when same-entity fusion is empty or missing value vars. - Add simple_avet_predicate_rows fast path with tighter AVET bounds and direct row collection for predicate/range queries. - Expand test_datahike_queries.ml to all 15 benchmark queries at size=2000. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
The AVET range path used fold_stored_bounded with an attr-only upper bound, which kept scanning until the next attribute and decoded every datom twice (stop check plus callback). slice_seq also materialized the full range into a list before iteration. Add fold_stored_avet_value_range to seek at the attr+value key prefix, stop on attr change or upper value using key parsing, and route AVET value-range slices through it. Expose fold_index_range for streaming query execution and remove the small-db AEVT array workaround. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- Store empty LMDB values for AVET keys and decode keys without re-parsing attr - Reuse a long-lived read transaction for index cursor scans - Keep merged-index AVET range queries on the LMDB bounded fold path - Skip redundant in_range checks and post-filter branches on exact bounds - Label compare benchmark output with storage backend (memory vs LMDB temp) - Add count_avet and query_profile probes for predicate scan diagnostics Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Use benchmark_memory_storage for Datahike compare benches: shared in-memory LMDB with MDB_NOSYNC/NOMETASYNC/WRITEMAP, materialized attr indexes via refresh_db_indexes after bulk load, and AVET array-range fast paths when avet_by_attr is populated. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Route AEVT exact lookups through cached aevt_by_attr arrays instead of LMDB slices. Optimize simple_same_entity_constant_rows with entity-set intersection, lazy per-entity value lookup for small result sets, and materialized tables only for large multi-attr scans. Benchmark harness: default repeats=2, warmup/sample=200ms, configurable JIT warmup, BENCH_SIZE for Datahike side, and FULL=1 for publication timing. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Support --query / BENCH_QUERY so one query can be timed quickly without running the full suite. Applies to the OCaml bench, shared Datahike bench, and compare_ocaml_datahike.sh (second positional arg or env). Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Remove the guard that skipped the fast path whenever value variables were present, which forced q2-q4 through the generic query engine. Use Hashtbl entity-id intersection and fill value tables via datoms/ AEVT attr scans so attrs like last-name are not missed by the slice-based primary_attr_datoms helper. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Implement dedicated query shapes for cross-entity value joins, or-join with constant name branches, and not-join with a single constant clause. These avoid the generic binding engine and scan indexes directly. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
fold_slice and find_first_slice now recognize structurally equal attr-only prefix bounds instead of requiring physical (==) equality on bound datoms. When that fast path was missed, fold_stored_bounded used full datom compare against e=0 bounds and filtered out every real entity datom. Also route datascript primary_attr_datoms through Db.primary_attr_datoms (Index.fold_attr_prefix with pending overlay and view), remove the duplicate Index.slice implementation, and restore join fast paths to use the fixed attr cache directly. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Recognize the Datahike benchmark pattern (follow ?e1 ?e2) with a single non-recursive follows rule and enumerate follows edges via primary_attr_datoms instead of the generic rule engine. Parity: q-rule still returns 667 rows @ size=2000. Benchmark @2000: ~0.011 ms OCaml vs ~0.22 ms Datahike (~0.05x). Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Take 100 datoms from mid-entity seek instead of the entity's exact prefix (which only had 5 facts). Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Support --sizes with default 50k+1M, batch entity builds, and raise LMDB mapsize so million-entity envs fit under no_subdir. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Drop the 1M default in favor of two large-but-cheaper stress sizes. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
cursor
Bot
changed the title
Route q through Query_impl only. Drop duplicated AVET/same-entity/OR/NOT/rule micro-paths and the unused entity_ids_array helper so the relation interpreter is the single fallback beside the planner. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Replace stub plan types with LEntityJoin/LScan/LFilter/LUnion/LAntiJoin, lower to physical ops with readiness-aware cost order, and order eligible where-clauses from the plan. Keep source order when any not is present so DataScript unbound-var errors stay observable; relational interpreter remains the permanent fallback. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Add the alignment plan and refresh the ADR plus comparison notes to match the live compile → relation-ops → interpreter pipeline. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Use AEVT point lookup for cardinality-one value attrs, fold single-pattern not-join into same-entity exclusion, and evaluate selective or-join unions on the relation path before age/value probes. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Port Datahike-like same-entity dense/binary-search AEVT multi-attr gather, cross-entity value hash-probe (q5), specialized AVET predicate rows with unique_rows, and aevt_attr_array scans for not/not-join into query_where. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Restore value-attr pattern order for specialized [e; attrs...] rows, defer candidate_entities until needed, and skip redundant filters on single-constant dense AVET id gathers. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Avoid list conversion on dense constant gathers, unroll 1/2/4-attr specialized rows, and scan not/not-join value attrs via reverse aevt array iteration. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- OrJoin constant branches: AVET entity-id union + AEVT value probe (no per-branch relation eval) - Defer (max_e+1) constant bitsets; AVET-fast NOT exclusion marking - AEVT array scan for [?e :attr ?v] patterns (q-rule follows) without Seq→list - Always unwrap Ref values via result_of_pattern_position on AEVT emit paths - Fix OrJoin relation merge: project each branch to join vars before union Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Try same-entity fusion, single-pattern AEVT scan, cross-entity join, and selective OrJoin on source-order clauses before calling plan_ordered_clauses. Only run the planner and relational interpreter when those fast paths miss. Also route single-clause [?e :attr ?v] queries through AEVT array scan before the generic interpreter. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Add single-pattern rule AEVT scan in eval_relation_rows (q-rule), early NOT+value-var AEVT scan in relation_of_same_entity_patterns, unrolled attr_count row emit for misaligned constant dense gather, and skip redundant entity_allowed checks when AVET ids already filter. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Use per-entity pull for ref_target_pull_relation when target set is small (<=512) instead of scanning all datoms. Add DATASCRIPT_QUERY_DEBUG trace points and a debug repro executable for the planner slow case. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Port pre-removal aligned_constant_rows kernel into relation_of_same_entity_patterns before source_context setup (attr_count <= 2). Short-circuit eval_relation_rows via same_entity_fused_relation. Remove List.rev from aligned dense AVET loops. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Add try_const_arr_aligned_rows inside rows_from_dense_aevt_gather using the constant attr array as alignment reference (matching pre-removal kernel). Fix value_var order in relation-level early dense path (List.rev). Enable early dense path for all attr counts when alignment succeeds. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Introduce try_fast_empty_relation_rows to handle same-entity dense gather and NOT AEVT scans before relation_of_same_entity_patterns setup. Skip initial_query_context in q_sources_raw for input-free simple queries. Benchmarks (size=2000): q2 ~0.009s, q-5-merge ~0.046s, q-not ~0.025s. Parity tests remain green (17/17). Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Clone replikativ/datahike under _deps/ for local reference. Document phase-by-phase mapping, per-bench-query behavior, and refactor plan to replace query_where special cases with query_exec fused execute. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- Extend OpEntityGroup IR with scan/merges/anti_scans/filters (Datahike shape) - Add impl/query_exec.ml: fused entity-group execute for single OpEntityGroup/OpScan - Wire query_api: compile -> Query_exec.run -> relational fallback - Remove try_fast_empty_relation_rows and ~1500 lines of duplicate dense kernels - Simplify relation_of_same_entity_patterns to hash_join fallback only - Route only plan_is_fused_execute shapes through execute; OR/NOT-join keep fallback Parity: test_shared_queries 17/17 green. Bench @ size=2000: q2 0.011, q-5-merge 0.055, q-or-join 0.051, q-rule 0.0095. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Replace dense-only gather special cases with drive-scan + AEVT forward-seek/dense index merges (lookupGE ≈ seekGE) and foldable NOT as anti-merge during the scan. Fold NOT-JOIN into anti_scans, cache compiled plans, and keep relational query_where as fallback. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Specialize q2 and q-5-merge dense emit, match try_not AEVT+bitset for NOT/NOT-JOIN, short-circuit single-op execute_plan, identity plan cache, and direct find projection when attrs already match. Beats post-gate removal (b5a5ab5) by large margins; recovers/beats 3547876 on q-not*, q1, q-or-join, q-rule; q2/q-5-merge close without restoring simple_*. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Avoid repeated resolve/normalize + avet lookup for the same ground attr/value within a bench run (q1/q2/q-5-merge drive scans). Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Reuse plan-identity kernels for q2 and q-5-merge, cache find-var names, and apply entity-group comparison filters in execute. Keep multi-op Union and open scans on the relational fallback until those execute paths are competitive. size=2000 recovers/beats 3547876 on q2 and q-5-merge. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Do not Hashtbl-key plans by query_clause list: structural hash/compare raises Invalid_argument when where clauses embed function bindings. Reuse the last plan via physical equality of the cached where list. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Unix.gettimeofday is unavailable under Melange/js_of_ocaml and broke CI linking for js_smoke and datascript_js. Route through the virtual Platform clock instead. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Expose last_query_exec_path and with_force_relation_fallback so tests can pin which engine ran. Assert hot shared shapes use fused execute and that forcing the relational fallback yields identical result digests. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Measure process RSS after build/query/tx/GC/close at 50k with each backend in an isolated process; compare against main's in-memory path. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Shared-API parity PR with Datahike-aligned query execute:
query_execlayer (analyze → logical → lower → execute + relational fallback)simple_*onqCI fixes
compile_planno longer Hashtbl-keys byquery_clause list(crashed on function bindings)debug_logusesPlatform.now_secondsinstead ofUnix.gettimeofday(Melange/js_of_ocaml unbound Unix / link failure)Query exec test coverage
test_query_exec_parity: asserts hot shapes (q1/q2/q3/q-5-merge/q-not*) takeFused_executeq-or/q-or-join/q-rule/qpred2) stay onRelation_fallbackQuery_plan.plan_is_fused_executeexported; same-entity plan test asserts fused shapeBench (size=2000)
Base:
main