Skip to content

MDEV-39834: Add TidesDB 9(v9.3.3) storage engine#5166

Open
guycipher wants to merge 4 commits into
MariaDB:11.4from
tidesdb:tidesdb-engine
Open

MDEV-39834: Add TidesDB 9(v9.3.3) storage engine#5166
guycipher wants to merge 4 commits into
MariaDB:11.4from
tidesdb:tidesdb-engine

Conversation

@guycipher
Copy link
Copy Markdown

The libtidesdb C library is vendored under storage/tidesdb/libtidesdb/ and compiled into a static archive that is statically linked into the loadable plugin module, mirroring storage/rocksdb, the engine is self-contained and needs no system-installed libtidesdb. include/providers is removed from the
engine's include dirs so the bundled zstd/lz4/snappy link directly instead of routing through MariaDB's compression-provider plugins.

For cross-version portability, spatial- and fulltext-index detection uses the HA_SPATIAL / HA_FULLTEXT key flags (with a HA_SPATIAL/HA_FULLTEXT -> *_legacy rename shim) rather than KEY::algorithm, which older servers leave
unset; my_global.h is included before handler.h so the server typedefs it needs are defined.

Adds the tidesdb mysql-test suite covering CRUD, MVCC and pessimistic locking, transactions/savepoints, isolation levels, compression, encryption, TTL, full-text, spatial, partitioning, object-store offload and crash/durability. The vector test auto-skips where the VECTOR type is unavailable (before 11.7).

Builds and full mysql-test suite shows green on MariaDB 11.4 (LTS) through to 13.0.2

The libtidesdb C library is vendored under storage/tidesdb/libtidesdb/ and
compiled into a static archive that is statically linked into the loadable
plugin module, mirroring storage/rocksdb, the engine is self-contained and
needs no system-installed libtidesdb. include/providers is removed from
the
engine's include dirs so the bundled zstd/lz4/snappy link directly instead
of routing through MariaDB's compression-provider plugins.

For cross-version portability, spatial- and fulltext-index detection uses
the HA_SPATIAL / HA_FULLTEXT key flags (with a HA_SPATIAL/HA_FULLTEXT ->
*_legacy rename shim) rather than KEY::algorithm, which older servers
leave
unset; my_global.h is included before handler.h so the server typedefs it
needs are defined.

Adds the tidesdb mysql-test suite covering CRUD, MVCC and
pessimistic locking, transactions/savepoints, isolation levels,
compression, encryption, TTL, full-text, spatial, partitioning, object-store offload
and crash/durability. The vector test auto-skips where the VECTOR type is
unavailable (before 11.7).

Builds and full mysql-test suite shows green on MariaDB 11.4 (LTS) through to
13.0.2
@CLAassistant
Copy link
Copy Markdown

CLAassistant commented Jun 2, 2026

CLA assistant check
All committers have signed the CLA.

Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request integrates the TidesDB storage engine into MariaDB, adding the core handler implementation in ha_tidesdb.h, build configuration in CMakeLists.txt, vendored libtidesdb source files, and an extensive test suite. The reviewer provided several constructive suggestions, including moving the public header copy operation in CMakeLists.txt from configure time to build time to prevent stale headers during incremental builds. Additionally, the reviewer recommended using C++11 in-class member initializers for various member variables in ha_tidesdb.h to avoid uninitialized memory issues, and explicitly marking the ha_tidesdb destructor with override.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread storage/tidesdb/CMakeLists.txt
Comment thread storage/tidesdb/ha_tidesdb.h
Comment thread storage/tidesdb/ha_tidesdb.h
Comment thread storage/tidesdb/ha_tidesdb.h
Comment thread storage/tidesdb/ha_tidesdb.h
Comment thread storage/tidesdb/ha_tidesdb.h Outdated
@guycipher
Copy link
Copy Markdown
Author

Base (11.4) >> 11.5 >> 12.0 >> … >> 12.x >> 13.0 >> main

Can propagate forward to every newer branch automatically.

@guycipher
Copy link
Copy Markdown
Author

Full TideSQL reference: https://tidesdb.com/reference/tidesql/

@guycipher
Copy link
Copy Markdown
Author

I see

/home/buildbot/storage/tidesdb/ha_tidesdb.cc: In member function 'virtual int ha_tidesdb::write_row(const uchar*)':
/home/buildbot/storage/tidesdb/ha_tidesdb.cc:6694:28: error: structured bindings only available with '-std=c++17' or '-std=gnu++17' [-Werror=c++17-extensions]
 6694 |                 for (auto &[term, tf] : tf_map)
      |                            ^
/home/buildbot/storage/tidesdb/ha_tidesdb.cc: In member function 'virtual int ha_tidesdb::update_row(const uchar*, const uchar*)':
/home/buildbot/storage/tidesdb/ha_tidesdb.cc:7898:32: error: structured bindings only available with '-std=c++17' or '-std=gnu++17' [-Werror=c++17-extensions]
 7898 |                     for (auto &[term, tf] : old_tf)
      |                                ^
/home/buildbot/storage/tidesdb/ha_tidesdb.cc:7906:32: error: structured bindings only available with '-std=c++17' or '-std=gnu++17' [-Werror=c++17-extensions]
 7906 |                     for (auto &[term, tf] : new_tf)
      |                                ^
/home/buildbot/storage/tidesdb/ha_tidesdb.cc:7926:32: error: structured bindings only available with '-std=c++17' or '-std=gnu++17' [-Werror=c++17-extensions]
 7926 |                     for (auto &[term, old_cnt] : old_tf)
      |                                ^
/home/buildbot/storage/tidesdb/ha_tidesdb.cc:7936:32: error: structured bindings only available with '-std=c++17' or '-std=gnu++17' [-Werror=c++17-extensions]
 7936 |                     for (auto &[term, new_cnt] : new_tf)
      |                                ^
/home/buildbot/storage/tidesdb/ha_tidesdb.cc: In member function 'virtual int ha_tidesdb::delete_row(const uchar*)':
/home/buildbot/storage/tidesdb/ha_tidesdb.cc:8206:28: error: structured bindings only available with '-std=c++17' or '-std=gnu++17' [-Werror=c++17-extensions]
 8206 |                 for (auto &[term, tf] : tf_map)
      |                            ^
/home/buildbot/storage/tidesdb/ha_tidesdb.cc: In member function 'virtual ha_rows ha_tidesdb::multi_range_read_info_const(uint, RANGE_SEQ_IF*, void*, uint, uint*, uint*, ha_rows, Cost_estimate*)':
/home/buildbot/storage/tidesdb/ha_tidesdb.cc:8609:25: error: 'struct TABLE' has no member named 'part_info'
 8609 |     if (table && table->part_info) return rows;
      |                         ^~~~~~~~~
/home/buildbot/storage/tidesdb/ha_tidesdb.cc: In member function 'virtual FT_INFO* ha_tidesdb::ft_init_ext(uint, uint, String*)':
/home/buildbot/storage/tidesdb/ha_tidesdb.cc:9850:16: error: structured bindings only available with '-std=c++17' or '-std=gnu++17' [-Werror=c++17-extensions]
 9850 |     for (auto &[pk_str, score] : doc_scores)
      |                ^
cc1plus: all warnings being treated as errors
make[2]: *** [storage/tidesdb/CMakeFiles/tidesdb.dir/build.make:79: storage/tidesdb/CMakeFiles/tidesdb.dir/ha_tidesdb.cc.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:10332: storage/tidesdb/CMakeFiles/tidesdb.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

Having a look as I'm gonna guess we need to pass that wf.

@guycipher
Copy link
Copy Markdown
Author

C++11 is base I am going to assume, 13 seems to be C++17. I'll keep the engine at bay with C++11 as the library aligns with C11.

Build fixes for MariaDB 11.4-11.7 (C++11; buildbot builds with -Werror):
- Replace C++17 structured bindings with C++11 iterator access in the
  FTS write_row/update_row/delete_row and ft_init_ext paths.
- Guard TABLE::part_info with WITH_PARTITION_STORAGE_ENGINE, which only
  defines that member when partitioning is compiled in.

Initialization hardening (PR review feedback):
- Value-initialize TidesDB_share encryption members (encrypted,
  encryption_key_id, encryption_key_version) in the constructor.
- Mark ~ha_tidesdb() override.
@guycipher
Copy link
Copy Markdown
Author

guycipher commented Jun 2, 2026

Something to have a look at: https://buildbot.mariadb.org/#/builders/554/builds/22266 and https://buildbot.mariadb.org/#/builders/554/builds/22266/steps/3/logs/stdio not sure if CI or something I need to touch on at plugin level.

@guycipher
Copy link
Copy Markdown
Author

Caught it, had to create mariadb-plugin-tidesdb.install

@guycipher
Copy link
Copy Markdown
Author

https://buildbot.mariadb.org/#/builders/534/builds/39281/steps/6/logs/stdio buildbot/amd64-ubuntu-2204-debug-ps does not seem to be due to TidesDB implementation. v9.3.3 of TidesDB and v4.5.4 of TideSQL plugin implemented. Ready for review.

@dr-m
Copy link
Copy Markdown
Contributor

dr-m commented Jun 3, 2026

https://buildbot.mariadb.org/#/builders/534/builds/39281/steps/6/logs/stdio buildbot/amd64-ubuntu-2204-debug-ps does not seem to be due to TidesDB implementation.

Right, there is an open ticket MDEV-38195 about failures of the test events.events_restart.

@gkodinov gkodinov added the External Contribution All PRs from entities outside of MariaDB Foundation, Corporation, Codership agreements. label Jun 3, 2026
@gkodinov gkodinov self-assigned this Jun 3, 2026
Copy link
Copy Markdown
Member

@gkodinov gkodinov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution! This is a preliminary review.

I'm approving it as it is, as there're no major violations. Or, if there are any, they're in the competency of the final reviewer. Nevertheless, I'll mention what I see that can be improved.

First of all: please consider merging some of the commits. Ideally there should be one commit per distinct "feature". I'd guess that makes 1 in your case.

Next: this technically is a new feature. As such, I'd suggest considering rebasing to the main branch. Unless of course, there is some special consideration why the targed should be as is.

Finally, there should be a Jira for house-keeping. I'll start one for you, using the PR explanation. But this needs to be improved significantly I'd guess, until it evolves into a proper design specification that would help with the final review.

There's also one small thing I've noticed below.

Please stand by for the final review.

Comment thread storage/tidesdb/plug.in Outdated
@gkodinov gkodinov changed the title Adds TidesDB 9(v9.3.3) storage engine MDEV-39834: Add TidesDB 9(v9.3.3) storage engine Jun 3, 2026
@gkodinov gkodinov requested a review from sanja-byelkin June 3, 2026 09:08
@gkodinov gkodinov assigned sanja-byelkin and unassigned gkodinov Jun 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

External Contribution All PRs from entities outside of MariaDB Foundation, Corporation, Codership agreements.

Development

Successfully merging this pull request may close these issues.

5 participants