feat(agent): hybrid BM25 search_wiki tool for query/chat agent - #234
Open
sebastianbraun25 wants to merge 1 commit into
Open
feat(agent): hybrid BM25 search_wiki tool for query/chat agent#234sebastianbraun25 wants to merge 1 commit into
sebastianbraun25 wants to merge 1 commit into
Conversation
Adds a dependency-free BM25 full-text index (openkb/fulltext_index.py) over concepts/entities/summaries pages, exposed as a new search_wiki tool alongside index.md-driven navigation in build_query_agent. Additive hybrid retrieval: surfaces pages whose one-line index summary omits a buried detail, without replacing existing navigation. Resolves VectifyAI#233.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
openkb query/openkb chatnavigate the wiki purely viaindex.mdone-line summaries plusLLM-directed page reads (
read_file,get_page_content). One-line summaries cannot surface factsburied deep in a page body (a specific figure, a niche keyword, an exact term). As a wiki grows,
the agent increasingly misses pages whose
index.mdsummary doesn't mention the exact term theuser asked about, even though the page body contains the answer.
Solution / Changes
openkb/fulltext_index.py: a dependency-free BM25 (Robertson/Sparck-Jones) indexover
concepts/,entities/, andsummaries/wiki pages (PAGE_CONTENT_DIRS), rebuiltin-memory on construction — cheap at the wiki sizes this pattern targets, no on-disk cache or
incremental update needed. Exposes
WikiFullTextIndex.search(query, top_k)returning rankedSearchHit(path, title, score, snippet)results.openkb/agent/tools.py: new plain functionsearch_wiki(query, wiki_root, top_k=5)formattingranked hits as
[[wikilink]] — title (score) / snippettext for the agent.openkb/agent/query.py: wiressearch_wikiin as a fourth tool (read_file,get_page_content,search_wiki,get_image) inbuild_query_agent— and therefore also inbuild_chat_agent, which extends the query agent's tools. Instructions updated with a newsearch-strategy step describing it as a hybrid, additive fallback: use alongside, not
instead of,
index.mdnavigation, so recall can only improve, never regress.Whoosh) isn't warranted, and
openkb's dependencies are pinned exactly and vetted deliberately(see
pyproject.toml).CLI flag, or schema changes.
Testing
tests/test_fulltext_index.py(new): tokenization/BM25 ranking,PAGE_CONTENT_DIRS-onlyscoping,
top_k, snippet extraction, title fallback, empty-index/empty-query edge cases.tests/test_agent_tools.py(newTestSearchWikiclass):search_wikiformatting, no-matchmessage,
top_k.tests/test_query.py(updated):build_query_agentnow exposes four tools includingsearch_wiki.ruff check,ruff format --check,mypy openkb,pytestall green (pre-existing,environment-specific failures unrelated to this change are unaffected).
Issues
Resolves #233