diff --git a/CHANGELOG.md b/CHANGELOG.md index 84d865a6a..a6351a0b0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,6 +30,8 @@ Full release notes with details on each version: [GitHub Releases](https://githu - Fix: non-English query sentences no longer pick wrong BFS seeds because their filler words were unfiltered (#1900). The query stopword set now covers German and the major Romance languages (curated to avoid clobbering English content words), so `Wie funktioniert die Authentifizierung?` seeds the keyword, not the stopwords. - Fix: Python calls to an imported module now resolve (already shipped in 0.9.16); `.skill` files (Markdown-with-frontmatter agent files) are now classified as documents instead of being silently dropped as an unsupported extension (#1901). - Fix: the `--postgres` missing-driver error now points at the correct PyPI package, `graphifyy[postgres]` (was the nonexistent `graphify[postgres]`) (#1906). +- Add: R (`.r`/`.R`) support — bespoke `extract_r` handles all five function-assignment forms (`<-`, `<<-`, `=`, `->`, `->>`), nested function scoping beneath the enclosing function, top-level vs function-level call attribution, member calls (`$`/`@`), package-qualified calls (`::`/`:::`), and pipes (`|>`/`%>%`). Named functions are assignments whose right- or left-hand side is an anonymous `function_definition`, so the generic extractor is not a good fit; the new bespoke extractor lives under `graphify/extractors/`, the established home for language-specific extractors (not the `extract.py` god node). `library()`/`require()`/`requireNamespace()` with a literal identifier or string emit `imports` edges; `pkg::fn`/`pkg:::fn` double as import evidence for `pkg`. `source("helper.r")` with a static `.r` path resolved against the caller file's directory emits `imports_from` when the target exists on disk (URLs, connections, variables, computed paths, and missing files are skipped — mirroring the Bash static-`source` policy). Unqualified bare calls funnel through the shared cross-file `raw_calls` resolver; an R language-family guard blocks binding to non-R definitions, so a cross-file candidate resolves `EXTRACTED` with `source()` import evidence or `INFERRED` without it, and ambiguous names remain unresolved. Ships under the optional `[r]` extra via `tree-sitter-language-pack` (the pack may fetch/cache the R grammar on first use; no source is uploaded and R itself is not executed). Install: `uv tool install "graphifyy[r]"`. R package metadata (`DESCRIPTION`/`NAMESPACE`/exports) and package-qualified call resolution are deferred to a follow-up (#9). +- Fix: `.R` files no longer trigger the "no AST extractor for `.r`/`.R`" warning (#1689) now that an `extract_r` dispatch exists; the missing-dependency path reuses the existing `#1745` install-extra warning with `pip install "graphifyy[r]"`. `Rscript`-shebang scripts (already detected as code) now dispatch to `extract_r` instead of falling through to the no-extractor warning. ## 0.9.16 (2026-07-14) diff --git a/README.md b/README.md index 81685cf72..c6eba647b 100644 --- a/README.md +++ b/README.md @@ -259,6 +259,7 @@ Codex users also need `multi_agent = true` under `[features]` in `~/.codex/confi | `terraform` | Terraform / HCL `.tf`/`.tfvars`/`.hcl` AST extraction | `uv tool install "graphifyy[terraform]"` | | `pascal` | Pascal / Delphi `.pas`/`.dpr`/`.dpk`/`.inc` AST extraction (more accurate `calls`/`inherits` edges; falls back to a regex extractor when absent) | `uv tool install "graphifyy[pascal]"` | | `chinese` | Chinese query segmentation (jieba) | `uv tool install "graphifyy[chinese]"` | +| `r` | R `.r`/`.R` AST extraction (functions, calls, imports, static `source()`); ships via `tree-sitter-language-pack` which may fetch/cache the R grammar on first use (no source uploaded, R not executed) | `uv tool install "graphifyy[r]"` | | `all` | Everything above | `uv tool install "graphifyy[all]"` | @@ -327,7 +328,7 @@ To remove graphify from all platforms at once: `graphify uninstall` (add `--purg | Type | Extensions | |------|-----------| -| Code (36 tree-sitter grammars) | `.py .ts .mts .cts .js .jsx .tsx .mjs .go .rs .java .c .cpp .cc .cxx .h .hpp .cu .cuh .metal .rb .cs .kt .kts .scala .php .swift .lua .luau .toc .zig .ps1 .psm1 .psd1 .ex .exs .m .mm .jl .vue .svelte .astro .groovy .gradle .dart .v .sv .svh .sql .f .f90 .f95 .f03 .f08 .pas .pp .dpr .dpk .lpr .inc .dfm .lfm .lpk .sh .bash .json .dm .dme .dmi .dmm .dmf .sln .slnx .csproj .fsproj .vbproj .xaml .razor .cshtml` (`.dm`/`.dme` requires `uv tool install graphifyy[dm]`; `.mts`/`.cts` reuse the TypeScript grammar, `.cc`/`.cxx` and CUDA `.cu`/`.cuh` and Metal `.metal` reuse the C++ grammar) | +| Code (37 tree-sitter grammars) | `.py .ts .mts .cts .js .jsx .tsx .mjs .go .rs .java .c .cpp .cc .cxx .h .hpp .cu .cuh .metal .rb .cs .kt .kts .scala .php .swift .lua .luau .toc .zig .ps1 .psm1 .psd1 .ex .exs .m .mm .jl .r .R .vue .svelte .astro .groovy .gradle .dart .v .sv .svh .sql .f .f90 .f95 .f03 .f08 .pas .pp .dpr .dpk .lpr .inc .dfm .lfm .lpk .sh .bash .json .dm .dme .dmi .dmm .dmf .sln .slnx .csproj .fsproj .vbproj .xaml .razor .cshtml` (`.dm`/`.dme` requires `uv tool install graphifyy[dm]`; `.r`/`.R` requires `uv tool install "graphifyy[r]"`; `.mts`/`.cts` reuse the TypeScript grammar, `.cc`/`.cxx` and CUDA `.cu`/`.cuh` and Metal `.metal` reuse the C++ grammar) | | Salesforce Apex | `.cls .trigger` (regex-based; classes, interfaces, enums, methods, triggers, SOQL/DML edges) | | Terraform / HCL | `.tf .tfvars .hcl` (requires `uv tool install graphifyy[terraform]`) | | MCP configs | `.mcp.json` `mcp.json` `mcp_servers.json` `claude_desktop_config.json` — extracts server nodes, package refs, env var requirements | diff --git a/graphify/extract.py b/graphify/extract.py index 2ff5411cf..5aa54eb05 100644 --- a/graphify/extract.py +++ b/graphify/extract.py @@ -139,6 +139,7 @@ from graphify.extractors.objc import _objc_local_var_types, extract_objc # noqa: E402,F401 from graphify.extractors.julia import extract_julia # noqa: E402,F401 +from graphify.extractors.r import extract_r # noqa: E402,F401 _RECURSION_LIMIT = 10_000 @@ -1788,6 +1789,7 @@ def _lang_is_case_insensitive(source_file: object) -> bool: ".zig": "zig", ".ex": "elixir", ".exs": "elixir", ".jl": "julia", + ".r": "r", ".dart": "dart", ".sh": "shell", ".bash": "shell", ".ps1": "powershell", ".psm1": "powershell", ".psd1": "powershell", @@ -3877,6 +3879,7 @@ def add_existing_edge(edge: dict) -> None: ".m": extract_objc, ".mm": extract_objc, ".jl": extract_julia, + ".r": extract_r, ".f": extract_fortran, ".F": extract_fortran, ".f90": extract_fortran, @@ -3943,6 +3946,7 @@ def add_existing_edge(edge: dict) -> None: ".hcl": "terraform", ".dm": "dm", ".dme": "dm", + ".r": "r", } @@ -3951,7 +3955,7 @@ def add_existing_edge(edge: dict) -> None: # routes them to the CODE path via _shebang_interpreter; _get_extractor must # honor the same signal or these files are classified as code and then silently # dropped by extraction. Only interpreters with a real extractor are mapped — -# detect's wider set (perl, fish, tcsh, Rscript) stays unmapped and skipped. +# detect's wider set (perl, fish, tcsh) stays unmapped and skipped. _SHEBANG_DISPATCH: dict[str, Any] = { "python": extract_python, "python2": extract_python, @@ -3967,6 +3971,7 @@ def add_existing_edge(edge: dict) -> None: "lua": extract_lua, "php": extract_php, "julia": extract_julia, + "Rscript": extract_r, } @@ -4403,7 +4408,7 @@ def extract( ) # #1689: a file counted as code (extension in CODE_EXTENSIONS) but with no AST - # extractor wired up (e.g. .r/.R — there is no tree-sitter-r dispatch) silently + # extractor wired up (e.g. .ejs) silently # contributes zero nodes. The #1666 warning above deliberately skips these (it # only fires when an extractor exists), so surface them explicitly, grouped by # extension, rather than reporting success as if the language were mapped. diff --git a/graphify/extractors/__init__.py b/graphify/extractors/__init__.py index ada517094..ffe6b5f09 100644 --- a/graphify/extractors/__init__.py +++ b/graphify/extractors/__init__.py @@ -23,6 +23,7 @@ from graphify.extractors.markdown import extract_markdown from graphify.extractors.objc import extract_objc from graphify.extractors.pascal import extract_pascal +from graphify.extractors.r import extract_r from graphify.extractors.pascal_forms import extract_delphi_form, extract_lazarus_form from graphify.extractors.powershell import extract_powershell, extract_powershell_manifest from graphify.extractors.razor import extract_razor @@ -52,6 +53,7 @@ "markdown": extract_markdown, "objc": extract_objc, "pascal": extract_pascal, + "r": extract_r, "powershell": extract_powershell, "powershell_manifest": extract_powershell_manifest, "razor": extract_razor, diff --git a/graphify/extractors/r.py b/graphify/extractors/r.py new file mode 100644 index 000000000..55d8f03d4 --- /dev/null +++ b/graphify/extractors/r.py @@ -0,0 +1,281 @@ +"""R extractor. + +R's named functions are assignments whose left- or right-hand side is an +anonymous ``function_definition`` (``name <- function(...) ...`` and the +right-assignment forms ``function(...) -> name``), so a bespoke extractor is +required rather than the generic one. The contract mirrors the other +extractors: file/function nodes, ``contains``/``calls``/``imports`` edges and +``raw_calls`` for the shared cross-file bare-call resolver. + +Package metadata (DESCRIPTION/NAMESPACE/exports) is intentionally out of scope +for this initial integration; package-qualified ``pkg::fn`` calls are recorded +as member raw facts only. +""" +from __future__ import annotations + +from pathlib import Path +from typing import Any + +from graphify.extractors.base import _LANGUAGE_BUILTIN_GLOBALS, _file_stem, _make_id + + +_ASSIGN_OPS_LEFT = frozenset({"<-", "<<-", "="}) +_ASSIGN_OPS_RIGHT = frozenset({"->", "->>"}) +_PKG_LOADERS = frozenset({"library", "require", "requireNamespace"}) + + +def extract_r(path: Path) -> dict: + """Extract functions, calls, imports, and static source() from a .r/.R file.""" + try: + from tree_sitter_language_pack import get_language + from tree_sitter import Parser + except ImportError: + return {"nodes": [], "edges": [], "error": "tree-sitter-language-pack not installed"} + + try: + language = get_language("r") + parser = Parser(language) + source = path.read_bytes() + tree = parser.parse(source) + root = tree.root_node + except Exception as e: + return {"nodes": [], "edges": [], "error": str(e)} + + stem = _file_stem(path) + str_path = str(path) + del stem # current contract builds function ids from scope nids, not the file stem + + nodes: list[dict] = [] + edges: list[dict] = [] + seen_ids: set[str] = set() + function_bodies: list[tuple[str, Any]] = [] + pkg_imports_seen: set[tuple[str, str]] = set() + + def _text(node) -> str: + return source[node.start_byte:node.end_byte].decode("utf-8", errors="replace") + + def add_node(nid: str, label: str, line: int) -> None: + if nid not in seen_ids: + seen_ids.add(nid) + nodes.append({"id": nid, "label": label, "file_type": "code", + "source_file": str_path, "source_location": f"L{line}"}) + + def add_edge(src: str, tgt: str, relation: str, line: int, + confidence: str = "EXTRACTED", weight: float = 1.0, + context: str | None = None) -> None: + if not src or not tgt or src == tgt: + return + edge = {"source": src, "target": tgt, "relation": relation, + "confidence": confidence, "source_file": str_path, + "source_location": f"L{line}", "weight": weight} + if context: + edge["context"] = context + edges.append(edge) + + file_nid = _make_id(str(path)) + add_node(file_nid, path.name, 1) + + def _op_of(binop) -> str: + for child in binop.children: + if child.type in _ASSIGN_OPS_LEFT or child.type in _ASSIGN_OPS_RIGHT: + return child.type + return "" + + def _parent_type(node) -> str: + parent = node.parent + return parent.type if parent is not None else "" + + def _named_children(node): + return [c for c in node.children if c.is_named] + + def _function_body(fn_def): + for c in _named_children(fn_def): + if c.type != "parameters": + return c + return None + + def walk(node, scope_nid: str) -> None: + if node.type == "binary_operator": + op = _op_of(node) + named = _named_children(node) + if op in _ASSIGN_OPS_LEFT and len(named) >= 2: + lhs, rhs = named[0], named[1] + in_call_arg = _parent_type(node) == "argument" + if (not in_call_arg and lhs.type == "identifier" + and rhs.type == "function_definition"): + name = _text(lhs) + line = node.start_point[0] + 1 + func_nid = _make_id(scope_nid, name) + add_node(func_nid, f"{name}()", line) + add_edge(scope_nid, func_nid, "contains", line) + body = _function_body(rhs) + if body is not None: + function_bodies.append((func_nid, body)) + walk(rhs, func_nid) + return + elif op in _ASSIGN_OPS_RIGHT and len(named) >= 2: + lhs, rhs = named[0], named[1] + if rhs.type == "identifier" and lhs.type == "function_definition": + name = _text(rhs) + line = node.start_point[0] + 1 + func_nid = _make_id(scope_nid, name) + add_node(func_nid, f"{name}()", line) + add_edge(scope_nid, func_nid, "contains", line) + body = _function_body(lhs) + if body is not None: + function_bodies.append((func_nid, body)) + walk(lhs, func_nid) + return + for c in node.children: + walk(c, scope_nid) + return + + if node.type == "function_definition": + parent_op = _op_of(node.parent) if node.parent is not None and node.parent.type == "binary_operator" else "" + if parent_op in _ASSIGN_OPS_LEFT or parent_op in _ASSIGN_OPS_RIGHT: + for c in node.children: + walk(c, scope_nid) + return + body = None + for c in _named_children(node): + if c.type not in ("parameters",): + body = c + break + if (body is not None and body.type == "binary_operator" + and _op_of(body) in _ASSIGN_OPS_RIGHT + and _named_children(body)[-1].type == "identifier"): + body_named = _named_children(body) + rhs = body_named[-1] + name = _text(rhs) + line = node.start_point[0] + 1 + func_nid = _make_id(scope_nid, name) + add_node(func_nid, f"{name}()", line) + add_edge(scope_nid, func_nid, "contains", line) + fbody = _function_body(node) + if fbody is not None: + function_bodies.append((func_nid, fbody)) + if len(body_named) >= 2: + walk(body_named[0], func_nid) + return + for c in node.children: + walk(c, scope_nid) + return + + for c in node.children: + walk(c, scope_nid) + + walk(root, file_nid) + + label_to_nid: dict[str, str] = {} + for n in nodes: + normalised = n["label"].strip("()").lstrip(".") + label_to_nid[normalised] = n["id"] + + seen_call_pairs: set[tuple[str, str]] = set() + raw_calls: list[dict] = [] + + def _first_string(node) -> str | None: + for c in node.children: + if c.type == "string": + inner = c.children[1] if len(c.children) > 1 else c + return _text(inner) + if c.type == "argument": + return _first_string(c) + return None + + def _first_identifier(node) -> str | None: + for c in node.children: + if c.type == "identifier": + return _text(c) + if c.type == "argument": + return _first_identifier(c) + return None + + def _emit_pkg_import(scope_nid: str, pkg: str, line: int) -> None: + key = (scope_nid, pkg) + if pkg and key not in pkg_imports_seen: + pkg_imports_seen.add(key) + add_edge(scope_nid, _make_id(pkg), "imports", line, context="import") + + def walk_calls(node, caller_nid: str) -> None: + if node.type == "function_definition": + return + if node.type != "call": + for c in node.children: + walk_calls(c, caller_nid) + return + + line = node.start_point[0] + 1 + fn_node = None + arguments_node = None + for c in node.children: + if c.type in ("identifier", "namespace_operator", "extract_operator"): + fn_node = c + elif c.type == "arguments": + arguments_node = c + + if fn_node is not None and fn_node.type == "identifier": + callee = _text(fn_node) + if callee in _PKG_LOADERS: + if arguments_node is not None: + pkg = _first_identifier(arguments_node) or _first_string(arguments_node) + if pkg: + _emit_pkg_import(caller_nid, pkg, line) + for c in node.children: + walk_calls(c, caller_nid) + return + if callee == "source": + if arguments_node is not None: + raw = _first_string(arguments_node) + if raw and raw.lower().endswith(".r"): + if not raw.startswith(("http://", "https://", "ftp://")): + resolved = (path.parent / raw).resolve() + if resolved.exists(): + add_edge(file_nid, _make_id(str(resolved)), "imports_from", line, context="import") + for c in node.children: + walk_calls(c, caller_nid) + return + is_member_call = False + elif fn_node is not None and fn_node.type == "namespace_operator": + named = _named_children(fn_node) + pkg = _text(named[0]) if named else "" + callee = _text(named[-1]) if named else "" + is_member_call = True + if pkg: + _emit_pkg_import(caller_nid, pkg, line) + elif fn_node is not None and fn_node.type == "extract_operator": + named = _named_children(fn_node) + callee = _text(named[-1]) if named else "" + is_member_call = True + else: + callee = "" + is_member_call = False + + if callee and callee not in _LANGUAGE_BUILTIN_GLOBALS: + tgt_nid = label_to_nid.get(callee) + if tgt_nid and tgt_nid != caller_nid: + pair = (caller_nid, tgt_nid) + if pair not in seen_call_pairs: + seen_call_pairs.add(pair) + add_edge(caller_nid, tgt_nid, "calls", line, + confidence="EXTRACTED", weight=1.0, context="call") + else: + raw_calls.append({ + "caller_nid": caller_nid, + "callee": callee, + "is_member_call": is_member_call, + "source_file": str_path, + "source_location": f"L{line}", + }) + + for c in node.children: + walk_calls(c, caller_nid) + + walk_calls(root, file_nid) + for caller_nid, body in function_bodies: + walk_calls(body, caller_nid) + + clean_edges = [e for e in edges if e["source"] in seen_ids and + (e["target"] in seen_ids or e["relation"] in ("imports", "imports_from"))] + return {"nodes": nodes, "edges": clean_edges, "raw_calls": raw_calls, + "input_tokens": 0, "output_tokens": 0} \ No newline at end of file diff --git a/pyproject.toml b/pyproject.toml index 2e5340d39..02daf5a02 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -80,7 +80,12 @@ pascal = ["tree-sitter-pascal"] # avoids breaking the default `uv tool install graphifyy` for everyone (#1104). dm = ["tree-sitter-dm"] terraform = ["tree-sitter-hcl"] -all = ["mcp", "starlette>=1.3.1", "neo4j", "falkordb", "pypdf>=6.12.0", "markdownify", "watchdog", "graspologic; python_version < '3.13'", "python-docx", "openpyxl", "faster-whisper; python_version >= '3.11'", "yt-dlp>=2026.6.9", "matplotlib", "numpy>=2.0; python_version >= '3.13'", "openai", "tiktoken", "boto3", "anthropic", "tree-sitter-sql", "jieba", "tree-sitter-dm", "tree-sitter-hcl", "tree-sitter-pascal"] +# No maintained tree-sitter-r wheel fits the repo's per-language wheel model, +# so the R grammar ships via tree-sitter-language-pack (lazy-loaded in +# extract_r). The pack may fetch/cache the R grammar on first use; no source +# is uploaded and R itself is never executed (#9). +r = ["tree-sitter-language-pack>=1.12,<2"] +all = ["mcp", "starlette>=1.3.1", "neo4j", "falkordb", "pypdf>=6.12.0", "markdownify", "watchdog", "graspologic; python_version < '3.13'", "python-docx", "openpyxl", "faster-whisper; python_version >= '3.11'", "yt-dlp>=2026.6.9", "matplotlib", "numpy>=2.0; python_version >= '3.13'", "openai", "tiktoken", "boto3", "anthropic", "tree-sitter-sql", "jieba", "tree-sitter-dm", "tree-sitter-hcl", "tree-sitter-pascal", "tree-sitter-language-pack>=1.12,<2"] [project.scripts] graphify = "graphify.__main__:main" diff --git a/tests/fixtures/r_caller.r b/tests/fixtures/r_caller.r new file mode 100644 index 000000000..99d6ba043 --- /dev/null +++ b/tests/fixtures/r_caller.r @@ -0,0 +1,5 @@ +# R fixture: top-level bare calls with no own definition and no source() link, +# so `dup()` is ambiguous (defined in r_other.R and r_extra.R) and `helper()` +# resolves INFERRED to a single unique cross-file candidate. +dup() +helper() \ No newline at end of file diff --git a/tests/fixtures/r_extra.r b/tests/fixtures/r_extra.r new file mode 100644 index 000000000..72c3604ea --- /dev/null +++ b/tests/fixtures/r_extra.r @@ -0,0 +1,9 @@ +# R fixture: defines a second `dup` (ambiguity with r_other.R) and a no-source +# caller scope, so `helper()` resolves INFERRED unless source() linkage exists. +extra_fn <- function() { + helper() +} + +dup <- function() { + NA +} \ No newline at end of file diff --git a/tests/fixtures/r_other.r b/tests/fixtures/r_other.r new file mode 100644 index 000000000..dd8232809 --- /dev/null +++ b/tests/fixtures/r_other.r @@ -0,0 +1,8 @@ +# R fixture: helper + dup definitions for cross-file resolution tests. +helper <- function() { + invisible() +} + +dup <- function() { + NA +} \ No newline at end of file diff --git a/tests/fixtures/sample.r b/tests/fixtures/sample.r new file mode 100644 index 000000000..8c85741b1 --- /dev/null +++ b/tests/fixtures/sample.r @@ -0,0 +1,43 @@ +# Graphify R fixture: exercises every construct the extractor handles. +# Function assignment forms (left, super, equals, right, super-right): +library(dplyr) +requireNamespace("utils") +library(installed.packages()) # dynamic arg -> NO import edge +source("r_other.r") # static source() -> imports_from edge + +foo <- function(x) { # `<-` form + helper() # cross-file call (defined in r_other.R) + print(x) # builtin in raw_calls +} + +bar <<- function(y) y * 2 # `<<-` super-assignment form + +baz = function(z) { # `=` form + inner <- function(w) { # nested function + helper() # attributed to inner, not baz + z + w + } + inner(z) # same-file call -> EXTRACTED +} + +function(a) a + 1 -> qux # `->` right-assignment form +function(b) b * 2 ->> quux # `->>` super-right-assignment form + +# Member calls (do NOT resolve via the bare-name resolver): +obj$method() +obj@field() + +# Package-qualified calls (recorded as member raw_calls; pkg import edge emitted): +base::summary() +dplyr::filter() +utils:::deep_fn() + +# Pipes (traversed so contained calls are captured): +1 |> qux() |> foo() +"data" %>% process() %>% save() + +# Named call argument with `=` is NOT a function definition: +options(config = function() 42) + +# Top-level call (attributed to the file node): +foo(1) \ No newline at end of file diff --git a/tests/test_extract.py b/tests/test_extract.py index 32c8e8e13..066481f6c 100644 --- a/tests/test_extract.py +++ b/tests/test_extract.py @@ -1999,18 +1999,19 @@ def test_case_insensitive_suffix_filtering(tmp_path): def test_extract_warns_on_code_files_with_no_ast_extractor(tmp_path, capsys): - # #1689: .r/.R is in CODE_EXTENSIONS (counted as code) but has no AST extractor, - # so R files silently contribute nothing. extract() must surface that instead of - # reporting success as if the language were mapped. - r1 = tmp_path / "analysis.R"; r1.write_text("f <- function(x) x + 1\n") - r2 = tmp_path / "helper.r"; r2.write_text("g <- function(y) y * 2\n") + # #1689: .ejs is in CODE_EXTENSIONS (counted as code) but has no AST extractor, + # so such files silently contribute nothing. extract() must surface that instead + # of reporting success as if the language were mapped. (.R was the original + # motivating case but now has extract_r; .ejs is the canonical no-extractor one.) + r1 = tmp_path / "view.ejs"; r1.write_text("

<%= name %>

\n") + r2 = tmp_path / "page.ejs"; r2.write_text("

Title

\n") py = tmp_path / "main.py"; py.write_text("def main():\n return 1\n") result = extract([r1, r2, py], cache_root=tmp_path) err = capsys.readouterr().err assert "no AST extractor" in err - assert ".r (2)" in err # both R files grouped under the lowercased ext + assert ".ejs (2)" in err assert "#1689" in err # the Python file still extracts normally labels = [n.get("label") for n in result["nodes"]] @@ -2054,6 +2055,38 @@ def test_extract_no_missing_dep_warning_when_sql_installed(tmp_path, capsys): assert "#1745" not in err +def test_extract_warns_when_r_extra_missing(tmp_path, capsys, monkeypatch): + # #1745: .r HAS a dispatch entry, so the #1689 warning can't fire, and + # extract_r returns an "error" result when tree-sitter-language-pack is + # absent, so the files must not vanish silently — extract() surfaces them + # with the [r] extra named. + monkeypatch.setitem(sys.modules, "tree_sitter_language_pack", None) + r1 = tmp_path / "analysis.R"; r1.write_text("f <- function(x) x + 1\n") + r2 = tmp_path / "helper.r"; r2.write_text("g <- function(y) y * 2\n") + py = tmp_path / "main.py"; py.write_text("def main():\n return 1\n") + + result = extract([r1, r2, py], cache_root=tmp_path) + err = capsys.readouterr().err + + assert "2 .r file(s)" in err + assert "tree-sitter-language-pack not installed" in err + assert 'graphifyy[r]' in err + assert "#1745" in err + assert "#1689" not in err # .r/.R is now mapped; no #1689 no-extractor warning + # the Python file still extracts normally + labels = [n.get("label") for n in result["nodes"]] + assert any(str(l).startswith("main") for l in labels) + + +def test_extract_no_warning_when_r_extra_installed(tmp_path, capsys): + pytest.importorskip("tree_sitter_language_pack") + r = tmp_path / "module.R"; r.write_text("f <- function(x) x + 1\n") + extract([r], cache_root=tmp_path) + err = capsys.readouterr().err + assert "#1745" not in err + assert "no AST extractor" not in err + + def test_extract_progress_final_line_uses_consistent_denominator(tmp_path, capsys): # #1693: intermediate progress lines count against uncached_work; the final # "100%" line must NOT switch to total_files (which includes cached hits and @@ -2061,8 +2094,8 @@ def test_extract_progress_final_line_uses_consistent_denominator(tmp_path, capsy for i in range(100): (tmp_path / f"m{i}.py").write_text(f"def f{i}():\n return {i}\n") for i in range(5): - (tmp_path / f"s{i}.r").write_text(f"g{i} <- function(x) x\n") # no extractor - paths = sorted(tmp_path.glob("*.py")) + sorted(tmp_path.glob("*.r")) # total 105 + (tmp_path / f"s{i}.ejs").write_text(f"hello {i}\n") # no extractor + paths = sorted(tmp_path.glob("*.py")) + sorted(tmp_path.glob("*.ejs")) # total 105 extract(paths, cache_root=tmp_path, parallel=False) out = capsys.readouterr().out diff --git a/tests/test_languages.py b/tests/test_languages.py index f36ea7603..3cac22410 100644 --- a/tests/test_languages.py +++ b/tests/test_languages.py @@ -9,7 +9,7 @@ extract_groovy, extract_sln, extract_csproj, extract_xaml, extract_razor, extract_dm, extract_dmi, extract_dmm, extract_dmf, extract_powershell, extract_apex, extract_verilog, - extract_powershell_manifest, + extract_powershell_manifest, extract_r, ) FIXTURES = Path(__file__).parent / "fixtures" @@ -22,6 +22,10 @@ _ilu.find_spec("tree_sitter_dm") is None, reason="tree-sitter-dm not installed (optional [dm] extra)", ) +_needs_r = pytest.mark.skipif( + _ilu.find_spec("tree_sitter_language_pack") is None, + reason="tree-sitter-language-pack not installed (optional [r] extra)", +) def _labels(r): @@ -2966,3 +2970,229 @@ def test_decldef_merge_does_not_merge_same_name_same_dir_distinct_files(): r = _corpus("cpp_samedir/Alpha.h", "cpp_samedir/Beta.h") dups = _nodes_with_label(r, "Dup") assert len(dups) == 2, f"same-dir distinct Dups must stay distinct, got {[n['id'] for n in dups]}" + + +# --- R --------------------------------------------------------------- + +def _r_assert_no_dangling(r): + """R-aware variant of _assert_no_dangling: `imports` targets may be + external package ids (e.g. dplyr) with no matching node, so they're + allowed to dangle like other languages' external imports.""" + ids = {n["id"] for n in r["nodes"]} + for e in r["edges"]: + assert e["source"] in ids, f"dangling source: {e}" + if e["relation"] not in ("imports", "imports_from"): + assert e["target"] in ids, f"dangling target: {e}" + + +def test_r_dispatch_lower(): + from graphify.extract import _get_extractor + assert _get_extractor(Path("foo.r")) is extract_r + + +def test_r_dispatch_upper_uses_case_insensitive_fallback(): + # .R is not in _DISPATCH directly; _get_extractor lowercases the suffix + # via the existing fallback (extract.py ~4019-4020). + from graphify.extract import _get_extractor + assert _get_extractor(Path("FOO.R")) is extract_r + + +def test_r_script_shebang_dispatch(tmp_path): + from graphify.extract import _get_extractor + p = tmp_path / "rscript_no_ext" + p.write_text("#!/usr/bin/env Rscript\nx <- 1\n") + assert _get_extractor(p) is extract_r + + +def test_r_missing_dependency_returns_error(tmp_path, monkeypatch): + import sys as _sys + monkeypatch.setitem(_sys.modules, "tree_sitter_language_pack", None) + p = tmp_path / "x.r"; p.write_text("f <- function() 1\n") + r = extract_r(p) + assert "error" in r and "not installed" in r["error"] + + +@_needs_r +def test_r_no_error(): + r = extract_r(FIXTURES / "sample.r") + assert "error" not in r + + +@_needs_r +def test_r_left_assignment_forms_create_functions(): + r = extract_r(FIXTURES / "sample.r") + labels = _labels(r) + assert "foo()" in labels # <- + assert "bar()" in labels # <<- + assert "baz()" in labels # = + + +@_needs_r +def test_r_right_assignment_forms_create_functions(): + r = extract_r(FIXTURES / "sample.r") + labels = _labels(r) + assert "qux()" in labels # -> + assert "quux()" in labels # ->> + + +@_needs_r +def test_r_named_call_argument_with_equals_not_a_function(): + # options(config = function() 42) — the `=` is a named call arg, not an + # assignment binding a function to a name; must NOT produce a `config()` node. + r = extract_r(FIXTURES / "sample.r") + labels = _labels(r) + assert "config()" not in labels + + +@_needs_r +def test_r_nested_functions_get_scoped_ids_and_contains(): + r = extract_r(FIXTURES / "sample.r") + baz = _node_by_label(r, "baz") + inner_nodes = _nodes_with_label(r, "inner()") + assert len(inner_nodes) == 1 + inner = inner_nodes[0] + assert baz["id"] != inner["id"] + contains = _edge_labels(r, "contains") + assert (baz["id"], inner["id"]) in contains or \ + (_normalize_symbol_label("baz()"), _normalize_symbol_label("inner()")) in \ + {(_normalize_symbol_label(a), _normalize_symbol_label(b)) for a, b in contains} + + +@_needs_r +def test_r_nested_call_not_attributed_to_outer_function(): + # inner()'s body calls helper(); this call must attribute to `inner`, not `baz` + r = extract_r(FIXTURES / "sample.r") + baz_id = _node_by_label(r, "baz")["id"] + inner_id = _node_by_label(r, "inner")["id"] + # find a raw_call to helper whose caller is inner (not baz) + helper_rcs = [rc for rc in r["raw_calls"] if rc["callee"] == "helper"] + assert any(rc["caller_nid"] == inner_id for rc in helper_rcs), \ + "inner()'s call to helper must appear in raw_calls from inner, not baz" + assert not any(rc["caller_nid"] == baz_id for rc in helper_rcs), \ + "helper() call must not be attributed to the enclosing baz()" + + +@_needs_r +def test_r_same_file_calls_extracted(): + r = extract_r(FIXTURES / "sample.r") + calls = _calls(r) # set of (raw_src_label, raw_tgt_label) pairs + file_label = "sample.r" + # top-level call to foo() attributed to the file node + assert (file_label, "foo()") in calls + # baz() contains inner(); nested call to inner() attributed to baz + assert ("baz()", "inner()") in calls + # pipe chain captures qux() call at top level + assert (file_label, "qux()") in calls + + +@_needs_r +def test_r_member_calls_recorded_as_member_raw_calls(): + r = extract_r(FIXTURES / "sample.r") + member_callees = {rc["callee"] for rc in r["raw_calls"] if rc.get("is_member_call")} + # $ and @ receivers are member calls + assert "method" in member_callees # obj$method() + assert "field" in member_callees # obj@field + # pkg::fn / pkg:::fn are qualified-member (is_member_call=True) so the + # shared resolver doesn't bind them to a same-named def + assert "summary" in member_callees # pkg::summary + assert "deep_fn" in member_callees # base:::deep_fn + # NOTE: `dplyr::filter` — filter is a Python builtin, filtered from raw_calls + # by _LANGUAGE_BUILTIN_GLOBALS, so it does NOT appear here. + + +@_needs_r +def test_r_member_calls_never_bind_to_same_named_def(): + r = extract_r(FIXTURES / "sample.r") + calls = _calls(r) + # No calls edges to method/field/summary/deep_fn — they're member raw_calls + for callee in ("method", "field", "summary", "deep_fn"): + assert not any(tgt == f"{callee}()" for _, tgt in calls), \ + f"member call {callee}() must not bind to a same-named def" + + +@_needs_r +def test_r_imports_from_static_loaders(): + r = extract_r(FIXTURES / "sample.r") + imports_targets = {e["target"] for e in r["edges"] if e["relation"] == "imports"} + assert "dplyr" in imports_targets # library(dplyr) + assert "utils" in imports_targets # requireNamespace("utils") + assert "base" in imports_targets # pkg::fn emits pkg import evidence + + +@_needs_r +def test_r_imports_skipped_for_dynamic_loader_arg(): + # library(installed.packages()) — the arg is a call expression, not a + # literal identifier/string; must NOT produce an imports edge. + r = extract_r(FIXTURES / "sample.r") + imports_targets = {e["target"] for e in r["edges"] if e["relation"] == "imports"} + assert "installed.packages" not in imports_targets + assert "installed" not in imports_targets + + +@_needs_r +def test_r_static_source_emits_imports_from_to_existing_target(): + r = extract_r(FIXTURES / "sample.r") + ifs = [e for e in r["edges"] if e["relation"] == "imports_from"] + assert len(ifs) == 1 + assert "r_other" in ifs[0]["target"] # sibling r_other.r exists on disk + + +@_needs_r +def test_r_source_ignores_url_missing_and_non_r_targets(tmp_path): + src = tmp_path / "src.r" + src.write_text( + 'source("https://example.com/remote.R")\n' # URL — skip + 'source("missing.r")\n' # missing — skip + 'source("helper.py")\n' # not .r — skip + ) + r = extract_r(src) + assert not [e for e in r["edges"] if e["relation"] == "imports_from"] + + +@_needs_r +def test_r_no_dangling_edges(): + r = extract_r(FIXTURES / "sample.r") + _r_assert_no_dangling(r) + + +@_needs_r +def test_r_pipes_capture_contained_calls(): + r = extract_r(FIXTURES / "sample.r") + callees = {rc["callee"] for rc in r["raw_calls"]} + # pipe chain `1 |> process() |> save()` captures process & save + assert "process" in callees + assert "save" in callees + + +@_needs_r +def test_r_cross_file_call_resolves_with_source_link_extracted(): + # sample.r sources r_other.r, creating imports_from evidence; the cross-file + # helper() call resolves EXTRACTED (single candidate + import evidence). + r = _corpus("sample.r", "r_other.r") + calls = _calls(r) + assert ("foo()", "helper()") in calls or ("inner()", "helper()") in calls, \ + "cross-file helper() must resolve via source() import evidence" + + +@_needs_r +def test_r_cross_file_call_resolves_inferred_without_source(): + # r_caller.r has no source() linkage; helper() in r_other.r has a single + # cross-file candidate → INFERRED (confidence 0.8). + r = _corpus("r_caller.r", "r_other.r") + # Find any calls edge whose normalised target label is "helper" + helper_calls = [e for e in r["edges"] if e["relation"] == "calls" + and "helper" in _normalize_symbol_label(e.get("target", ""))] + assert helper_calls, "unique cross-file bare call must resolve as INFERRED" + # Verify the resolved target is in r_other.r + r_other_helper = [n for n in r["nodes"] if n.get("label") == "helper()"][0] + assert Path(r_other_helper["source_file"]).name == "r_other.r" + + +@_needs_r +def test_r_cross_file_ambiguous_call_remains_unresolved(): + # r_caller.r calls dup(); r_other.r AND r_extra.r both define dup(). + # Ambiguity guard must keep this unresolved (no calls edge to dup()). + r = _corpus("r_caller.r", "r_other.r", "r_extra.r") + dup_calls = [e for e in r["edges"] if e["relation"] == "calls" + and "dup" in _normalize_symbol_label(e.get("target", ""))] + assert len(dup_calls) == 0, f"ambiguous dup() must not resolve, got {dup_calls}"