feat(parsers): initialize language grammars lazily (#68)

[vitali87]

Problem

Issue #68: a repo in one language paid for all 14 grammars anyway. load_parsers() imported every grammar module and compiled every language's full query set (functions, classes, calls, imports, locals, highlights, plus two combined queries) eagerly, and it did all of that again on EVERY call: the CLI, the MCP server, the file editor, and each of the test suite's hundreds of load_parsers() calls each recompiled the world.

Fix

• load_parsers() now returns two lazy views over a process-wide store. The views subclass dict, so every dict[SupportedLanguage, ...]-typed consumer signature keeps working: a missing key routes through __missing__ and triggers a one-time grammar load for exactly that language; __contains__ and get (which must be overridden because dict.get bypasses __missing__ in C) probe on demand. Full-view operations (iteration, len, keys/values/items) load every spec'd language first, so no consumer ever observes a partial availability picture, and an all-languages probe that finds nothing still raises NO_LANGUAGES.
• Grammar module imports moved from import time of parser_loader into a per-language cached resolver, so merely importing the module no longer touches any tree_sitter_* package.
• StructureProcessor collected package_indicators by iterating queries.values(), which would have forced every grammar to load; it now reads the static LANGUAGE_SPECS directly (grammar-independent data). Side effect: package detection now covers all spec'd languages even when a grammar is unavailable, which is the more correct behavior.
• The store is a process singleton: repeated load_parsers() calls share loaded parsers instead of recompiling. _reset_parser_cache() exists as a test hook.

Validation

• Startup: load_parsers() alone drops from roughly 0.6 to 0.9s (eager, main) to interpreter-startup cost (about 0.3 to 0.5s total wall, nothing loaded); with one language actually used it stays around 0.4s. Per-language cost is now paid only for languages present in the repo.
• Full suite: 5441 passed with only the two documented memgraph-integration environmental flakes. Suite wall time dropped from 8m46s to 4m39s on the same machine, since workers stopped recompiling all query sets per load_parsers() call.
• The two extra skips vs older runs are the Milvus optional-dependency skips introduced by feat(vector-store): add optional Milvus backend #780, unrelated to this change.

Tests

RED -> GREEN: test_lazy_parser_loading.py — deferral (nothing compiles until first use, touching python does not load rust), on-demand membership (lang in parsers loads that one language truthfully), process-level sharing across load_parsers() calls, unknown-language absence, and full-view completeness (iteration covers every available language).

[gemini-code-assist]

Code Review

This pull request introduces lazy loading for tree-sitter language parsers and queries to optimize startup performance, ensuring grammars are loaded and cached only on first use. Feedback from the reviewer highlights that the current implementation is not thread-safe for concurrent environments. Specifically, the reviewer recommends importing the threading module and adding locking mechanisms to prevent race conditions during grammar loading in _ensure and when initializing the global _store in load_parsers().

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[greptile-apps]

Greptile Summary

This PR changes parser initialization so language grammars load only when needed. The main changes are:

• Adds a process-wide lazy grammar store behind load_parsers().
• Returns read-only Mapping views for parsers and queries instead of eager dictionaries.
• Caches per-language grammar imports and compiled query sets after first use.
• Updates parser consumers to accept Mapping inputs.
• Uses static language specs for package indicators so structure scanning does not load every grammar.
• Adds tests for deferred loading, membership checks, shared cache behavior, concurrent first loads, failed-load cleanup, and full-view completeness.

Confidence Score: 5/5

Safe to merge with minimal risk.

The lazy-loading implementation is covered by focused tests for deferral, cache sharing, concurrent access, partial-load windows, failed-load cleanup, and full-view behavior. The previous concurrency and partial-load issues are addressed by requiring both parser and query entries before treating a language as loaded. The remaining changes are narrow type updates from dict to Mapping or static-spec lookup changes.

No files require special attention.

T-Rex Logs

What T-Rex did

• Pytest collection completed but reported eight setup errors due to AttributeError: module 'codebase_rag' has no attribute 'cli' in codebase_rag/tests/conftest.py:97.
• Ran the runtime-validation harness; it exited with code 0 and showed an empty after-parser grammar state and stable Python-parser state (after_parser_loader_import_grammar_modules [], after_load_parsers_grammar_modules [], rust_module_loaded_after_python_access False, same_python_parser_across_calls True, full_view_keys_equal True).
• Saved the harness source used for runtime validation for inspection.

_{Ran code and verified through T-Rex}

Important Files Changed

Filename	Overview
codebase_rag/parser_loader.py	Introduces process-wide lazy grammar loading with cached imports, shared parser/query views, locking, and partial-load cleanup.
codebase_rag/parsers/structure_processor.py	Derives package indicators from static language specs to avoid forcing lazy grammar loads during structure detection.
codebase_rag/tests/test_lazy_parser_loading.py	Adds coverage for deferred grammar compilation, on-demand membership, cache sharing, concurrent loading, partial-load protection, and full-view completeness.
codebase_rag/graph_updater.py	Updates parser and query parameters to accept read-only Mapping views.
codebase_rag/parsers/import_processor.py	Updates import processing to accept lazy query mappings while preserving existing membership and indexing behavior.
codebase_rag/tests/test_file_editor.py	Adjusts file editor initialization expectations for lazy parser mappings and on-demand language loading.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant Consumer
participant Loader as load_parsers()
participant Store as _LazyGrammarStore
participant View as _LazyLanguageView
participant Grammar as tree_sitter grammar

Consumer->>Loader: request parsers, queries
Loader->>Store: create/reuse process-wide store
Loader-->>Consumer: return lazy Mapping views
Consumer->>View: access language key / membership
View->>Store: _ensure(language)
alt language not loaded
    Store->>Grammar: import module and compile queries
    Grammar-->>Store: parser + LanguageQueries
    Store-->>View: cached language available
else already loaded
    Store-->>View: reuse cached parser + queries
end
View-->>Consumer: parser/query value or absence

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant Consumer
participant Loader as load_parsers()
participant Store as _LazyGrammarStore
participant View as _LazyLanguageView
participant Grammar as tree_sitter grammar

Consumer->>Loader: request parsers, queries
Loader->>Store: create/reuse process-wide store
Loader-->>Consumer: return lazy Mapping views
Consumer->>View: access language key / membership
View->>Store: _ensure(language)
alt language not loaded
    Store->>Grammar: import module and compile queries
    Grammar-->>Store: parser + LanguageQueries
    Store-->>View: cached language available
else already loaded
    Store-->>View: reuse cached parser + queries
end
View-->>Consumer: parser/query value or absence

Loading

_{Reviews (4): Last reviewed commit: "style(parsers): collapse the twin-entry ..." | Re-trigger Greptile}

[vitali87]

@greptile review

[vitali87]

@greptile review

[vitali87]

@greptile review

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
96.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud