From 87d8f0e048516057d5c3f1813b97e506f2df9801 Mon Sep 17 00:00:00 2001 From: Juan Ezquerro LLanes Date: Mon, 13 Jul 2026 15:05:58 +0200 Subject: [PATCH 1/5] refactor: modularize configuration system and implement safe atomic file I/O - Split internal/config/env.go into dedicated config.go, io.go, and resolve.go - Implement ReadFileSafely and atomic file write system via temp files for security - Refactor engine components to use new config structures - Add CI workflow and cleanup outdated wiki pages --- .code-reducer.yaml | 68 ++-- .github/workflows/ci.yml | 26 ++ .github/workflows/release.yml | 12 +- .gitignore | 16 +- AGENTS.md | 9 - README.md | 20 +- cmd/root.go | 12 +- cmd/setup.go | 79 ++-- examples/go/.code-reducer.yaml | 48 +++ examples/terraform/.code-reducer.yaml | 55 +++ internal/config/config.go | 75 ++++ internal/config/env.go | 223 ----------- internal/config/io.go | 83 ++++ internal/config/resolve.go | 109 ++++++ internal/engine/cache.go | 33 +- internal/engine/chunking.go | 80 ++-- internal/engine/client.go | 67 ++-- internal/engine/constants.go | 17 + internal/engine/json_parser.go | 93 +---- internal/engine/orchestrator.go | 141 ++++--- internal/engine/runner.go | 36 +- internal/engine/synthesize.go | 78 ++-- internal/engine/tree.go | 5 +- internal/engine/utils.go | 12 +- internal/security/errors.go | 10 + internal/security/security.go | 51 ++- internal/tools/file_tools.go | 81 ++-- wiki/.metadata.json | 89 ----- wiki/architecture.md | 187 --------- wiki/modules/cmd.md | 255 +++++------- wiki/modules/internal.md | 541 -------------------------- wiki/modules/internal_config.md | 121 ------ wiki/modules/internal_engine.md | 276 ------------- wiki/modules/internal_security.md | 137 ------- wiki/modules/internal_tools.md | 110 ------ wiki/modules/root.md | 307 --------------- wiki/quickstart.md | 189 --------- 37 files changed, 978 insertions(+), 2773 deletions(-) create mode 100644 .github/workflows/ci.yml delete mode 100644 AGENTS.md create mode 100644 examples/go/.code-reducer.yaml create mode 100644 examples/terraform/.code-reducer.yaml create mode 100644 internal/config/config.go delete mode 100644 internal/config/env.go create mode 100644 internal/config/io.go create mode 100644 internal/config/resolve.go create mode 100644 internal/engine/constants.go create mode 100644 internal/security/errors.go delete mode 100644 wiki/.metadata.json delete mode 100644 wiki/architecture.md delete mode 100644 wiki/modules/internal.md delete mode 100644 wiki/modules/internal_config.md delete mode 100644 wiki/modules/internal_engine.md delete mode 100644 wiki/modules/internal_security.md delete mode 100644 wiki/modules/internal_tools.md delete mode 100644 wiki/modules/root.md delete mode 100644 wiki/quickstart.md diff --git a/.code-reducer.yaml b/.code-reducer.yaml index a8bd6d5..012bd73 100644 --- a/.code-reducer.yaml +++ b/.code-reducer.yaml @@ -2,6 +2,26 @@ model_id: ornith:9b ollama_base_url: http://localhost:11434 ollama_num_ctx: 20000 docs_dir: wiki + +system_prompt: | + You are Code-Reducer, an expert technical writer and code analyzer. Your job is to strictly follow instructions. You do not yap, you do not write filler. + DEFENSIVE RULES: 1. Do NOT use absolute terms ('always', 'never', 'zero') unless explicitly proven. 2. Do NOT guess downstream consequences or invent unhandled paths. If an error is swallowed, just say it is swallowed. 3. Do NOT name standard library packages unless explicitly stated in the source text. 4. Only report facts you are 100% sure about. + +module_synthesis_prompt: |- + Task: Write a technical documentation page for a code module based on the provided list of its internal components. + Rule 1: Group related functions and classes under appropriate Markdown headings. + Rule 2: Explain the responsibility of the module and the data flow. + Rule 3: Keep it highly technical and dense. + +architecture_prompt: |- + Task: Write a global architecture or quickstart document based on the module summaries. + Rule 1: Explain the system boundaries and how the modules interact. + Rule 2: Provide a dense, developer-friendly overview. + +file_fact_consolidation_prompt: |- + You are a specialized code documentation assistant. + Consolidate, deduplicate and merge the following facts extracted from different chunks of the same file into a single, cohesive summary. + extraction_steps: - name: API_SIGNATURES prompt: |- @@ -19,45 +39,11 @@ extraction_steps: prompt: |- Task: Analyze side effects and error handling. Output: Detail how this code communicates with the outside world (I/O like network, disk, DB) and how it handles/returns errors (wrap, sentinel, panic). + ignore: - - .git - - node_modules - - bower_components - - dist - - build - - cache - - __pycache__ - - .pytest_cache - - .mypy_cache - - .tox - - venv - - .venv - - code-reducer - - .gemini - - .code-reducer.yaml - - .code-reducer.lock - - AGENTS.md - - README.md - - go.sum - - .gitignore -ignore_extensions: - - .png - - .jpg - - .jpeg - - .gif - - .pdf - - .exe - - .dll - - .so - - .o - - .a - - .zip - - .gz - - .tar - - .lock - - .pyc - - .pyo - - .pyd - - -lock.json - - .lock.yaml - - pnpm-lock.yaml + - README.md + - .code-reducer.yaml + - go.sum + - go.mod + - screeshots + - examples diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..5e0fb81 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,26 @@ +name: CI Pipeline + +on: + push: + branches: [ main, master ] + pull_request: + branches: [ main, master ] + +jobs: + build-and-test: + name: Build & Test + runs-on: ubuntu-latest + steps: + - name: Checkout Repository + uses: actions/checkout@v4 + + - name: Set up Go + uses: actions/setup-go@v5 + with: + go-version: '1.26' + + - name: Run Vet + run: go vet ./... + + - name: Run Tests with Race Detector + run: go test -race -v ./... diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index cb9e61d..fe3e80f 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -20,13 +20,19 @@ jobs: goarch: "386" steps: - name: Checkout Repository - uses: actions/checkout@v7 + uses: actions/checkout@v4 - name: Set up Go - uses: actions/setup-go@v6 + uses: actions/setup-go@v5 with: go-version: '1.26' + - name: Run Vet + run: go vet ./... + + - name: Run Tests + run: go test -race -v ./... + - name: Build Binary env: GOOS: ${{ matrix.goos }} @@ -55,7 +61,7 @@ jobs: fi - name: Upload to Release - uses: softprops/action-gh-release@v3 + uses: softprops/action-gh-release@v2 with: files: ${{ steps.package.outputs.asset_path }} env: diff --git a/.gitignore b/.gitignore index 6c0f780..00611d9 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,16 @@ .code-reducer.lock -code-reducer \ No newline at end of file +code-reducer + +# IDEs and Editors +.idea/ +.vscode/ +*.suo +*.ntvs* +*.njsproj +*.sln +*.swp + +# Build output +dist/ +tmp/ +*.tmp.* \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md deleted file mode 100644 index 1780c8a..0000000 --- a/AGENTS.md +++ /dev/null @@ -1,9 +0,0 @@ -# AI Agent Guidelines - -This repository contains automatically generated documentation under the wiki directory to help AI coding agents understand the system architecture, design patterns, and module structure: - -- **System Blueprint**: Refer to wiki/architecture.md for a high-level system overview, module relationships, and boundary definitions. -- **Developer Quickstart**: Refer to wiki/quickstart.md for onboarding steps, coding patterns, and configuration settings. -- **Module Details**: Explore wiki/modules/ for directory-level summaries and API descriptions of internal packages. - -Before making changes, analyze these files to align with existing design choices and code structures. diff --git a/README.md b/README.md index 546c8f5..69eb8f7 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ Designed specifically for **local development and private LLMs**, Code-Reducer u ## πŸƒ Quick Start ### Prerequisites -* **Go**: Version 1.21 or higher. +* **Go**: Version 1.26 or higher. * **Ollama**: Running locally with a compatible model downloaded (e.g., `ornith:9b` or `gemma4:26b`). ### 1. Build from Source (or Download Release) @@ -107,7 +107,7 @@ extraction_steps: ``` ### Precedence Order -Code-Reducer implements a four-tier configuration resolution chain (defined in `internal/config/env.go`): +Code-Reducer implements a four-tier configuration resolution chain (defined in `internal/config/resolve.go`): ``` [1. CLI Overrides] ──► [2. Environment Variables] ──► [3. YAML Config File] ──► [4. System Defaults] @@ -199,22 +199,19 @@ To serialize execution across multiple terminal windows or background jobs, the 2. **PID Recording**: Writes the current Process ID (PID) to the lockfile. 3. **Git Isolation**: The runner automatically checks if `.code-reducer.lock` is ignored. If not, it safely appends it to the project's `.gitignore` file. -#### TOCTOU Symlink Hijacking Defense (`WriteFileSafely`) -When writing documentation files to the filesystem: -1. **Open Without Truncation**: Omit `O_TRUNC` on initial open (`os.OpenFile`) to prevent truncating a target file if the path has been replaced with a symlink. -2. **Symlink Verification**: Perform `os.Lstat` on the target path to verify that it is not a symbolic link. -3. **Descriptor Validation**: Obtain stats on the open file descriptor (`f.Stat()`) and compare it with the `Lstat` results via `os.SameFile()`. If the inodes do not match, a TOCTOU (Time-of-Check to Time-of-Use) symlink replacement race is detected and the write operation is aborted. -4. **Safe Truncation & Write**: Only after validation is complete is the file safely truncated (`f.Truncate(0)`) and written. +#### TOCTOU Symlink Hijacking & Safe File I/O +1. **Safe Reading (`ReadFileSafely`)**: When reading files, the engine performs `os.Lstat` on the path to verify it is not a symbolic link. It then opens the file descriptor, calls `f.Stat()`, and compares it with the `Lstat` results via `os.SameFile()`. If the inodes do not match, a TOCTOU symlink replacement race is detected and the read operation is aborted. +2. **Atomic Writing (`WriteFileSafely` & `SaveConfig`)**: To prevent data corruption, all file writes are performed atomically. The engine creates a temporary file in the target directory (`os.CreateTemp`), writes the contents, calls `Sync()` to flush to disk, closes the descriptor, sets the permissions, and atomically replaces the target file via `os.Rename`. --- ### 3. File Discovery, Binary, and Ignore Filters Repository scanning is executed using `filepath.WalkDir` coupled with multiple layers of evaluation: -1. **Pruning Subtrees**: Common default dependency, cache, and build directories (`.git`, `node_modules`, `bower_components`, `dist`, `build`, `cache`, `__pycache__`, `venv`, `.venv`, and directory names ending in `.egg-info`) are skipped using `filepath.SkipDir` at the walk root, saving CPU cycles. +1. **Pruning Subtrees**: Directories that are dot-prefixed (such as `.git` or `.venv`), end in `.egg-info`, or match any ignore rules (from `.gitignore` or configuration) are skipped entirely using `filepath.SkipDir` during traversal, saving CPU cycles. 2. **Ignore Matching Rules**: Ignores loaded from the project's `.gitignore` and specified in the YAML configuration are merged and compiled using a dedicated Gitignore library (`go-gitignore`), ensuring 100% compliance with standard Git semantic rules (like negations and deep globs). 3. **Binary Classification (Null-Byte Scanner & Fast-Path)**: - * **Known Extensions**: Files with explicitly ignored extensions (e.g. `.png`, `.pdf`, `.zip`, `.exe`) or lockfile suffixes (`*-lock.json`, `pnpm-lock.yaml`) are ignored. + * **Filter Flow**: Files not matching the text extension allowlist (such as unknown suffixes) are checked via binary signature fallback. * **Text Fast-Path**: Common source code extensions (like `.go`, `.js`, `.py`, `.md`) are instantly classified as text, entirely bypassing I/O bottlenecks. * **Fallback Null-Byte Scan**: Unlabelled files are caught by checking the first `1024` bytes for a null byte (`0x00`). If a null byte is found, the file is classified as a binary and skipped. @@ -229,7 +226,7 @@ Instead of relying on external Git diff parsing during runtime, Code-Reducer imp * **Added**: File is present in the workspace but missing from the cache. * **Modified**: File is present in both, but its current SHA256 does not match the cached hash. * **Deleted**: File exists in the cache but is missing from the workspace. Deleted files are automatically pruned from the cache. -* **Caching & Metadata Cache (`.metadata.json`)**: The metadata cache maps file paths to their `SHA256` and generated list of facts, alongside a map of directory modules. During updates, the engine matches active files against the cache, garbage-collects cache entries for deleted files, and updates the `last_documented_commit` tracker to `"local"` upon a successful run. +* **Caching & Metadata Cache (`.metadata.json`)**: The metadata cache maps file paths to their `SHA256` and generated list of facts, alongside a map of directory modules. During updates, the engine matches active files against the cache and garbage-collects cache entries for deleted files. --- @@ -238,7 +235,6 @@ Instead of relying on external Git diff parsing during runtime, Code-Reducer imp * **HTTP Request Timeout**: Configured to `10 minutes` to handle complex summarizations. * **Ollama API Schema**: Communicates with the `/api/chat` POST endpoint. * **Fail-Fast Client**: The LLM client is strictly fail-fast and does not perform retry attempts or exponential backoffs when calling the Ollama service. Any failure immediately returns an error. -* **Stream Processing**: Fully supports streaming response chunks via `StreamLLM` using line-by-line streaming from the `/api/chat` endpoint and invoking a token-callback. However, the recursive Map-Reduce pipeline invokes the synchronous `CallLLM` method for processing stability. --- diff --git a/cmd/root.go b/cmd/root.go index 820023a..b4f3434 100644 --- a/cmd/root.go +++ b/cmd/root.go @@ -16,7 +16,7 @@ import ( ) var ( - modelIdFlag string + modelIDFlag string numCtxFlag string ) @@ -30,7 +30,7 @@ var RootCmd = &cobra.Command{ } func init() { - RootCmd.PersistentFlags().StringVar(&modelIdFlag, "model-id", "", "Specify LLM model ID") + RootCmd.PersistentFlags().StringVar(&modelIDFlag, "model-id", "", "Specify LLM model ID") RootCmd.PersistentFlags().StringVar(&numCtxFlag, "num-ctx", "", "Specify Ollama context window size") } @@ -59,7 +59,7 @@ func executeCommand(mode string) error { } // Resolve the merged configuration - cfg, err := config.ResolveConfig(repoRoot, modelIdFlag, numCtxFlag) + cfg, err := config.ResolveConfig(repoRoot, modelIDFlag, numCtxFlag) if err != nil { return err } @@ -81,10 +81,10 @@ func executeCommand(mode string) error { defer stop() runner := engine.NewRunner(cfg) - err = runner.Run(ctx, repoRoot, mode, func(ev engine.Event) { - if ev.Type == "status" { + err = runner.Run(ctx, repoRoot, engine.Mode(mode), func(ev engine.Event) { + if ev.Type == engine.EventStatus { fmt.Println(ev.Message) - } else if ev.Type == "error" { + } else if ev.Type == engine.EventError { fmt.Fprintf(os.Stderr, "Error: %s\n", ev.Message) } else { fmt.Println(ev.Message) diff --git a/cmd/setup.go b/cmd/setup.go index 7ca06e3..18b9acc 100644 --- a/cmd/setup.go +++ b/cmd/setup.go @@ -67,18 +67,16 @@ func RunSetupFlow(repoRoot string) error { } var customIgnores []string - var customExtensions []string if existingCfg != nil { - customIgnores = subtractSlice(existingCfg.Ignore, config.DefaultIgnores) - customExtensions = subtractSlice(existingCfg.IgnoreExtensions, config.DefaultIgnoredExtensions) + customIgnores = existingCfg.Ignore } - userInputIgnores, ignoresModified := promptStringList(reader, "Enter custom directories/files to ignore (comma-separated)", customIgnores) + userInputIgnores, ignoresModified := promptStringList(reader, "Enter directories, files, or patterns to ignore (comma-separated)", customIgnores) var ignores []string if ignoresModified { if len(userInputIgnores) > 0 { - ignores = config.MergeAndDeduplicate(config.DefaultIgnores, userInputIgnores) + ignores = userInputIgnores } else { ignores = []string{} } @@ -86,27 +84,11 @@ func RunSetupFlow(repoRoot string) error { if existingCfg != nil { ignores = existingCfg.Ignore } else { - ignores = config.DefaultIgnores - } - } - - userInputExtensions, extensionsModified := promptStringList(reader, "Enter custom file extensions to ignore (comma-separated)", customExtensions) - var ignoreExtensions []string - if extensionsModified { - if len(userInputExtensions) > 0 { - ignoreExtensions = config.MergeAndDeduplicate(config.DefaultIgnoredExtensions, userInputExtensions) - } else { - ignoreExtensions = []string{} - } - } else { - if existingCfg != nil { - ignoreExtensions = existingCfg.IgnoreExtensions - } else { - ignoreExtensions = config.DefaultIgnoredExtensions + ignores = []string{} } } - existingDocsDir := "wiki" + existingDocsDir := config.DefaultDocsDir if existingCfg != nil && existingCfg.DocsDir != "" { existingDocsDir = existingCfg.DocsDir } @@ -119,14 +101,37 @@ func RunSetupFlow(repoRoot string) error { extractionSteps = config.DefaultExtractionSteps } + existingSystemPrompt := config.DefaultSystemPrompt + existingModuleSynthesisPrompt := config.DefaultModuleSynthesisPrompt + existingArchitecturePrompt := config.DefaultArchitecturePrompt + existingFileFactConsolidationPrompt := config.DefaultFileFactConsolidationPrompt + + if existingCfg != nil { + if existingCfg.SystemPrompt != "" { + existingSystemPrompt = existingCfg.SystemPrompt + } + if existingCfg.ModuleSynthesisPrompt != "" { + existingModuleSynthesisPrompt = existingCfg.ModuleSynthesisPrompt + } + if existingCfg.ArchitecturePrompt != "" { + existingArchitecturePrompt = existingCfg.ArchitecturePrompt + } + if existingCfg.FileFactConsolidationPrompt != "" { + existingFileFactConsolidationPrompt = existingCfg.FileFactConsolidationPrompt + } + } + newCfg := &config.Config{ - ModelID: modelInput, - OllamaBaseURL: urlInput, - OllamaNumCtx: numCtx, - DocsDir: docsDirInput, - ExtractionSteps: extractionSteps, - Ignore: ignores, - IgnoreExtensions: ignoreExtensions, + ModelID: modelInput, + OllamaBaseURL: urlInput, + OllamaNumCtx: numCtx, + DocsDir: docsDirInput, + ExtractionSteps: extractionSteps, + Ignore: ignores, + SystemPrompt: existingSystemPrompt, + ModuleSynthesisPrompt: existingModuleSynthesisPrompt, + ArchitecturePrompt: existingArchitecturePrompt, + FileFactConsolidationPrompt: existingFileFactConsolidationPrompt, } err = config.SaveConfig(repoRoot, newCfg) @@ -191,16 +196,4 @@ func promptString(reader *bufio.Reader, promptMsg, existingVal string) string { return input } -func subtractSlice[T comparable](a, b []T) []T { - bMap := make(map[T]bool) - for _, item := range b { - bMap[item] = true - } - var result []T - for _, item := range a { - if !bMap[item] { - result = append(result, item) - } - } - return result -} + diff --git a/examples/go/.code-reducer.yaml b/examples/go/.code-reducer.yaml new file mode 100644 index 0000000..6b83427 --- /dev/null +++ b/examples/go/.code-reducer.yaml @@ -0,0 +1,48 @@ +# Example .code-reducer.yaml for analyzing Go projects +model_id: "ornith:9b" +ollama_base_url: "http://localhost:11434" +ollama_num_ctx: 8192 +docs_dir: "wiki" + +# System Prompt defines the AI persona +system_prompt: | + You are Code-Reducer, an expert technical writer and Go code analyzer. Your job is to strictly follow instructions. You do not yap, you do not write filler. + DEFENSIVE RULES: + 1. Do NOT use absolute terms ('always', 'never', 'zero') unless explicitly proven. + 2. Do NOT guess downstream consequences or invent unhandled paths. If an error is swallowed, just say it is swallowed. + 3. Do NOT name standard library packages unless explicitly stated in the source text. + 4. Only report facts you are 100% sure about. + +# Prompts for synthesized output +module_synthesis_prompt: | + Task: Write a technical documentation page for a code module based on the provided list of its internal components. + Rule 1: Group related functions and classes under appropriate Markdown headings. + Rule 2: Explain the responsibility of the module and the data flow. + Rule 3: Keep it highly technical and dense. + +architecture_prompt: | + Task: Write a global architecture or quickstart document based on the module summaries. + Rule 1: Explain the system boundaries and how the modules interact. + Rule 2: Provide a dense, developer-friendly overview. + +file_fact_consolidation_prompt: | + You are a specialized code documentation assistant. Consolidate, deduplicate and merge the following facts extracted from different chunks of the same file into a single, cohesive summary. + +# Custom extraction steps during map phase +extraction_steps: + - name: "API_SIGNATURES" + prompt: | + Task: Extract the public surface area of the file. + Output: A strict Markdown list of all exported structs, interfaces, and methods. For each, note the actual input and output types. Ignore internal logic. + - name: "BUSINESS_LOGIC" + prompt: | + Task: Analyze the business logic and domain concepts. + Output: Explain what business rules or domain concepts this file solves. Describe the high-level algorithmic flow. Ignore implementation details. + - name: "STATE_AND_CONCURRENCY" + prompt: | + Task: Analyze state mutation and concurrency. + Output: List all mutable state (global variables, changing struct fields) and what concurrency mechanisms (e.g., sync.Mutex, channels) protect them. If none, state 'No mutable state'. + - name: "ERRORS_AND_SIDE_EFFECTS" + prompt: | + Task: Analyze side effects and error handling. + Output: Detail how this code communicates with the outside world (I/O like network, disk, DB) and how it handles/returns errors (wrap, sentinel, panic). diff --git a/examples/terraform/.code-reducer.yaml b/examples/terraform/.code-reducer.yaml new file mode 100644 index 0000000..6b4e981 --- /dev/null +++ b/examples/terraform/.code-reducer.yaml @@ -0,0 +1,55 @@ +# Example .code-reducer.yaml for analyzing Terraform infrastructure projects +model_id: "ornith:9b" +ollama_base_url: "http://localhost:11434" +ollama_num_ctx: 8192 +docs_dir: "wiki" + +# Tell the LLM to behave as a Cloud Architect and IaC analyzer +system_prompt: | + You are Code-Reducer, an expert Cloud Architect and Terraform infrastructure analyzer. Your job is to strictly follow instructions. You do not yap, you do not write filler. + DEFENSIVE RULES: + 1. Do NOT assume resources exist unless explicitly declared in the configuration. + 2. Do NOT guess downstream resource side effects unless directly linked (e.g., via security group attachments or subnet associations). + 3. Only report resource definitions and configurations you are 100% sure about. + +# Synthesis prompt for directory modules (which represent infrastructure sub-components) +module_synthesis_prompt: | + Task: Write a technical documentation page for a Terraform module based on the provided list of its internal resources, variables, and sub-modules. + Rule 1: Group related resources (e.g., networking, compute, database, IAM) under appropriate Markdown headings. + Rule 2: Explain the primary responsibility of this module, the resource dependency flow, and security configurations. + Rule 3: Keep it highly technical, listing key resources, inputs, and outputs. + +# Synthesis prompt for the global infrastructure overview +architecture_prompt: | + Task: Write a global cloud architecture overview and deployment guide based on the module summaries. + Rule 1: Explain the high-level cloud topology, provider requirements, and how modules connect (e.g., VPC connecting to EKS and RDS). + Rule 2: Provide a dense, developer-friendly infrastructure overview and standard Terraform deployment commands (init, plan, apply). + +file_fact_consolidation_prompt: | + You are a specialized infrastructure documentation assistant. Consolidate, deduplicate and merge the following facts extracted from different chunks of the same Terraform file into a single, cohesive summary. + +# Custom extraction steps during map phase +extraction_steps: + - name: "PROVISIONED_RESOURCES" + prompt: | + Task: Extract resources and data sources defined in this file. + Output: A strict Markdown list of all resources (`resource "type" "name"`) and data sources (`data "type" "name"`). Note key parameters (e.g. instance type, VPC CIDR, backend configurations). + - name: "VARIABLES_AND_OUTPUTS" + prompt: | + Task: Analyze input variables, outputs, and local variables. + Output: List all input variables (with types and default values) and outputs exported by this file. Mention any complex local variables (`locals`) that define key configurations. + - name: "MODULE_DEPENDENCIES" + prompt: | + Task: Analyze sub-module usage. + Output: Identify any external or internal modules called in this file (`module "name"`), their source paths, and the key inputs passed to them. + - name: "IAM_AND_SECURITY" + prompt: | + Task: Analyze permissions, IAM roles, and security groups. + Output: Detail security groups, firewall rules, IAM roles, policies, and KMS keys defined in this file. Outline what access permissions or network paths they allow. + +# Ignore Terraform state, local plugins, and lock files +ignore: + - ".terraform" + - "*.tfstate" + - "*.tfstate.backup" + - ".terraform.lock.hcl" diff --git a/internal/config/config.go b/internal/config/config.go new file mode 100644 index 0000000..1e127ce --- /dev/null +++ b/internal/config/config.go @@ -0,0 +1,75 @@ +package config + +const ( + // CodeReducerModelIDEnvKey is the env key for model ID override. + CodeReducerModelIDEnvKey = "CODE_REDUCER_MODEL_ID" + // OllamaBaseURLEnvKey is the env key for Ollama URL override. + OllamaBaseURLEnvKey = "OLLAMA_BASE_URL" + // OllamaNumCtxEnvKey is the env key for context size override. + OllamaNumCtxEnvKey = "OLLAMA_NUM_CTX" + + // OllamaDefaultBaseURL is the default URL for local Ollama api. + OllamaDefaultBaseURL = "http://localhost:11434" + // OllamaDefaultModelID is the default LLM model. + OllamaDefaultModelID = "ornith:9b" + // OllamaDefaultNumCtx is the default context size. + OllamaDefaultNumCtx = 8192 + // DefaultDocsDir is the default documentation folder name. + DefaultDocsDir = "wiki" + // ConfigFileName is the configuration filename. + ConfigFileName = ".code-reducer.yaml" + configFilePerm = 0600 + + // DefaultSystemPrompt is the default system instructions for the LLM. + DefaultSystemPrompt = "You are Code-Reducer, an expert technical writer and code analyzer. Your job is to strictly follow instructions. You do not yap, you do not write filler.\n" + + "DEFENSIVE RULES: 1. Do NOT use absolute terms ('always', 'never', 'zero') unless explicitly proven. 2. Do NOT guess downstream consequences or invent unhandled paths. If an error is swallowed, just say it is swallowed. 3. Do NOT name standard library packages unless explicitly stated in the source text. 4. Only report facts you are 100% sure about.\n" + + // DefaultModuleSynthesisPrompt is the default prompt for folder summaries. + DefaultModuleSynthesisPrompt = "Task: Write a technical documentation page for a code module based on the provided list of its internal components.\nRule 1: Group related functions and classes under appropriate Markdown headings.\nRule 2: Explain the responsibility of the module and the data flow.\nRule 3: Keep it highly technical and dense." + + // DefaultArchitecturePrompt is the default prompt for architecture synthesis. + DefaultArchitecturePrompt = "Task: Write a global architecture or quickstart document based on the module summaries.\nRule 1: Explain the system boundaries and how the modules interact.\nRule 2: Provide a dense, developer-friendly overview." + + // DefaultFileFactConsolidationPrompt is the default prompt for consolidation. + DefaultFileFactConsolidationPrompt = "You are a specialized code documentation assistant.\nConsolidate, deduplicate and merge the following facts extracted from different chunks of the same file into a single, cohesive summary." +) + +// ExtractionStep represents a single fact-extraction phase for files. +type ExtractionStep struct { + Name string `yaml:"name"` + Prompt string `yaml:"prompt"` +} + +// Config represents the schema of .code-reducer.yaml +type Config struct { + ModelID string `yaml:"model_id"` + OllamaBaseURL string `yaml:"ollama_base_url"` + OllamaNumCtx int `yaml:"ollama_num_ctx"` + DocsDir string `yaml:"docs_dir"` + SystemPrompt string `yaml:"system_prompt"` + ModuleSynthesisPrompt string `yaml:"module_synthesis_prompt"` + ArchitecturePrompt string `yaml:"architecture_prompt"` + FileFactConsolidationPrompt string `yaml:"file_fact_consolidation_prompt"` + ExtractionSteps []ExtractionStep `yaml:"extraction_steps"` + Ignore []string `yaml:"ignore"` +} + +// DefaultExtractionSteps is the standard list of extraction steps. +var DefaultExtractionSteps = []ExtractionStep{ + { + Name: "API_SIGNATURES", + Prompt: "Task: Extract the public surface area of the file.\nOutput: A strict Markdown list of all exported structs, interfaces, and methods. For each, note the actual input and output types. Ignore internal logic.", + }, + { + Name: "BUSINESS_LOGIC", + Prompt: "Task: Analyze the business logic and domain concepts.\nOutput: Explain what business rules or domain concepts this file solves. Describe the high-level algorithmic flow. Ignore implementation details.", + }, + { + Name: "STATE_AND_CONCURRENCY", + Prompt: "Task: Analyze state mutation and concurrency.\nOutput: List all mutable state (global variables, changing struct fields) and what concurrency mechanisms (e.g., sync.Mutex, channels) protect them. If none, state 'No mutable state'.", + }, + { + Name: "ERRORS_AND_SIDE_EFFECTS", + Prompt: "Task: Analyze side effects and error handling.\nOutput: Detail how this code communicates with the outside world (I/O like network, disk, DB) and how it handles/returns errors (wrap, sentinel, panic).", + }, +} diff --git a/internal/config/env.go b/internal/config/env.go deleted file mode 100644 index 4df49e7..0000000 --- a/internal/config/env.go +++ /dev/null @@ -1,223 +0,0 @@ -package config - -import ( - "fmt" - "os" - "path/filepath" - "strconv" - - "gopkg.in/yaml.v3" -) - -const ( - CodeReducerModelIdEnvKey = "CODE_REDUCER_MODEL_ID" - OllamaBaseUrlEnvKey = "OLLAMA_BASE_URL" - OllamaNumCtxEnvKey = "OLLAMA_NUM_CTX" - - OllamaDefaultBaseURL = "http://localhost:11434" - OllamaDefaultModelID = "ornith:9b" - OllamaDefaultNumCtx = 8192 - ConfigFileName = ".code-reducer.yaml" -) - -type ExtractionStep struct { - Name string `yaml:"name"` - Prompt string `yaml:"prompt"` -} - -// Config represents the schema of .code-reducer.yaml -type Config struct { - ModelID string `yaml:"model_id"` - OllamaBaseURL string `yaml:"ollama_base_url"` - OllamaNumCtx int `yaml:"ollama_num_ctx"` - DocsDir string `yaml:"docs_dir"` - ExtractionSteps []ExtractionStep `yaml:"extraction_steps"` - Ignore []string `yaml:"ignore"` - IgnoreExtensions []string `yaml:"ignore_extensions"` -} - -// getConfigPath returns the absolute path to the configuration file. -func getConfigPath(cwd string) string { - return filepath.Join(cwd, ConfigFileName) -} - -// ConfigExists checks if .code-reducer.yaml exists in the specified directory. -func ConfigExists(cwd string) bool { - _, err := os.Stat(getConfigPath(cwd)) - return err == nil -} - -// LoadConfig reads and parses .code-reducer.yaml from the specified directory. -func LoadConfig(cwd string) (*Config, error) { - data, err := os.ReadFile(getConfigPath(cwd)) - if err != nil { - return nil, err - } - var cfg Config - if err := yaml.Unmarshal(data, &cfg); err != nil { - return nil, fmt.Errorf("failed to parse yaml config: %w", err) - } - return &cfg, nil -} - -// SaveConfig writes the configuration to .code-reducer.yaml in the specified directory. -func SaveConfig(cwd string, cfg *Config) error { - data, err := yaml.Marshal(cfg) - if err != nil { - return fmt.Errorf("failed to marshal yaml: %w", err) - } - if err := os.WriteFile(getConfigPath(cwd), data, 0600); err != nil { - return fmt.Errorf("failed to write config file: %w", err) - } - return nil -} - -var DefaultIgnores = []string{ - ".git", - "node_modules", - "bower_components", - "dist", - "build", - "cache", - "__pycache__", - ".pytest_cache", - ".mypy_cache", - ".tox", - "venv", - ".venv", - "code-reducer", - ".gemini", - "AGENTS.md", - ".code-reducer.yaml", - ".code-reducer.lock", -} - -var DefaultIgnoredExtensions = []string{ - ".png", - ".jpg", - ".jpeg", - ".gif", - ".pdf", - ".exe", - ".dll", - ".so", - ".o", - ".a", - ".zip", - ".gz", - ".tar", - ".lock", - ".pyc", - ".pyo", - ".pyd", - "-lock.json", - ".lock.yaml", - "pnpm-lock.yaml", -} - -var DefaultExtractionSteps = []ExtractionStep{ - { - Name: "API_SIGNATURES", - Prompt: "Task: Extract the public surface area of the file.\nOutput: A strict Markdown list of all exported structs, interfaces, and methods. For each, note the actual input and output types. Ignore internal logic.", - }, - { - Name: "BUSINESS_LOGIC", - Prompt: "Task: Analyze the business logic and domain concepts.\nOutput: Explain what business rules or domain concepts this file solves. Describe the high-level algorithmic flow. Ignore implementation details.", - }, - { - Name: "STATE_AND_CONCURRENCY", - Prompt: "Task: Analyze state mutation and concurrency.\nOutput: List all mutable state (global variables, changing struct fields) and what concurrency mechanisms (e.g., sync.Mutex, channels) protect them. If none, state 'No mutable state'.", - }, - { - Name: "ERRORS_AND_SIDE_EFFECTS", - Prompt: "Task: Analyze side effects and error handling.\nOutput: Detail how this code communicates with the outside world (I/O like network, disk, DB) and how it handles/returns errors (wrap, sentinel, panic).", - }, -} - -// MergeAndDeduplicate merges two slices and removes duplicates. -func MergeAndDeduplicate[T comparable](a, b []T) []T { - seen := make(map[T]bool) - var result []T - for _, item := range append(a, b...) { - if !seen[item] { - seen[item] = true - result = append(result, item) - } - } - return result -} - -// ResolveConfig merges CLI overrides, environment variables, YAML config, and system defaults. -// It returns a fully resolved Config struct ready to be used by the pipeline runner and LLM client. -func ResolveConfig(repoRoot, modelIdFlag, numCtxFlag string) (*Config, error) { - cfg, err := LoadConfig(repoRoot) - if err != nil { - if !os.IsNotExist(err) { - return nil, fmt.Errorf("failed to load configuration file: %w", err) - } - cfg = &Config{} - } - - // Deduplicate ignores: start with default ignores, then add user config ignores - resolvedIgnore := MergeAndDeduplicate(DefaultIgnores, cfg.Ignore) - - // Deduplicate extensions: start with default extensions, then add user config extensions - resolvedExtensions := MergeAndDeduplicate(DefaultIgnoredExtensions, cfg.IgnoreExtensions) - - // Extraction steps - resolvedSteps := cfg.ExtractionSteps - if len(resolvedSteps) == 0 { - resolvedSteps = DefaultExtractionSteps - } - - resolved := &Config{ - Ignore: resolvedIgnore, - IgnoreExtensions: resolvedExtensions, - ExtractionSteps: resolvedSteps, - } - - // 1. Resolve Model ID: Default > YAML > Env > Flag - resolved.ModelID = OllamaDefaultModelID - if cfg.ModelID != "" { - resolved.ModelID = cfg.ModelID - } - if envVal := os.Getenv(CodeReducerModelIdEnvKey); envVal != "" { - resolved.ModelID = envVal - } - if modelIdFlag != "" { - resolved.ModelID = modelIdFlag - } - - // 2. Resolve Ollama Base URL: Default > YAML > Env - resolved.OllamaBaseURL = OllamaDefaultBaseURL - if cfg.OllamaBaseURL != "" { - resolved.OllamaBaseURL = cfg.OllamaBaseURL - } - if envVal := os.Getenv(OllamaBaseUrlEnvKey); envVal != "" { - resolved.OllamaBaseURL = envVal - } - - // 3. Resolve Ollama Context Size: Default > YAML > Env > Flag - resolved.OllamaNumCtx = OllamaDefaultNumCtx - if cfg.OllamaNumCtx > 0 { - resolved.OllamaNumCtx = cfg.OllamaNumCtx - } - if envVal := os.Getenv(OllamaNumCtxEnvKey); envVal != "" { - if n, err := strconv.Atoi(envVal); err == nil && n > 0 { - resolved.OllamaNumCtx = n - } - } - if numCtxFlag != "" { - if n, err := strconv.Atoi(numCtxFlag); err == nil && n > 0 { - resolved.OllamaNumCtx = n - } - } - - // 4. Resolve DocsDir: Default > YAML - resolved.DocsDir = "wiki" - if cfg.DocsDir != "" { - resolved.DocsDir = cfg.DocsDir - } - - return resolved, nil -} diff --git a/internal/config/io.go b/internal/config/io.go new file mode 100644 index 0000000..46c4df9 --- /dev/null +++ b/internal/config/io.go @@ -0,0 +1,83 @@ +package config + +import ( + "fmt" + "os" + "path/filepath" + "strings" + + "gopkg.in/yaml.v3" +) + +// getConfigPath returns the absolute path to the configuration file. +func getConfigPath(cwd string) string { + return filepath.Join(cwd, ConfigFileName) +} + +// ConfigExists checks if .code-reducer.yaml exists in the specified directory. +func ConfigExists(cwd string) bool { + _, err := os.Stat(getConfigPath(cwd)) + return err == nil +} + +// LoadConfig reads and parses .code-reducer.yaml from the specified directory. +func LoadConfig(cwd string) (*Config, error) { + data, err := os.ReadFile(getConfigPath(cwd)) + if err != nil { + return nil, err + } + var cfg Config + if err := yaml.Unmarshal(data, &cfg); err != nil { + return nil, fmt.Errorf("failed to parse yaml config: %w", err) + } + return &cfg, nil +} + +// SaveConfig writes the configuration to .code-reducer.yaml in the specified directory. +func SaveConfig(cwd string, cfg *Config) error { + data, err := yaml.Marshal(cfg) + if err != nil { + return fmt.Errorf("failed to marshal yaml: %w", err) + } + + yamlStr := string(data) + replacements := []string{ + "\nsystem_prompt:", "\n\nsystem_prompt:", + "\nmodule_synthesis_prompt:", "\n\nmodule_synthesis_prompt:", + "\narchitecture_prompt:", "\n\narchitecture_prompt:", + "\nfile_fact_consolidation_prompt:", "\n\nfile_fact_consolidation_prompt:", + "\nextraction_steps:", "\n\nextraction_steps:", + "\nignore:", "\n\nignore:", + } + for i := 0; i < len(replacements); i += 2 { + yamlStr = strings.ReplaceAll(yamlStr, replacements[i], replacements[i+1]) + } + + tmpFile, err := os.CreateTemp(cwd, ConfigFileName+".tmp.*") + if err != nil { + return fmt.Errorf("failed to create temp file: %w", err) + } + tmpName := tmpFile.Name() + defer func() { + tmpFile.Close() + os.Remove(tmpName) + }() + + if _, err := tmpFile.Write([]byte(yamlStr)); err != nil { + return fmt.Errorf("failed to write config to temp file: %w", err) + } + if err := tmpFile.Sync(); err != nil { + return fmt.Errorf("failed to sync temp file: %w", err) + } + if err := tmpFile.Close(); err != nil { + return fmt.Errorf("failed to close temp file: %w", err) + } + if err := os.Chmod(tmpName, configFilePerm); err != nil { + return fmt.Errorf("failed to chmod temp file: %w", err) + } + + if err := os.Rename(tmpName, getConfigPath(cwd)); err != nil { + return fmt.Errorf("failed to rename temp config file: %w", err) + } + return nil +} diff --git a/internal/config/resolve.go b/internal/config/resolve.go new file mode 100644 index 0000000..efe85b5 --- /dev/null +++ b/internal/config/resolve.go @@ -0,0 +1,109 @@ +package config + +import ( + "fmt" + "os" + "strconv" +) + +// mergeAndDeduplicate merges two slices and removes duplicates. +func mergeAndDeduplicate[T comparable](a, b []T) []T { + seen := make(map[T]bool) + var result []T + for _, item := range append(a, b...) { + if !seen[item] { + seen[item] = true + result = append(result, item) + } + } + return result +} + +// ResolveConfig merges CLI overrides, environment variables, YAML config, and system defaults. +// It returns a fully resolved Config struct ready to be used by the pipeline runner and LLM client. +func ResolveConfig(repoRoot, modelIDFlag, numCtxFlag string) (*Config, error) { + cfg, err := LoadConfig(repoRoot) + if err != nil { + if !os.IsNotExist(err) { + return nil, fmt.Errorf("failed to load configuration file: %w", err) + } + cfg = &Config{} + } + + // Deduplicate ignores + resolvedIgnore := mergeAndDeduplicate(cfg.Ignore, nil) + + // Extraction steps + resolvedSteps := cfg.ExtractionSteps + if len(resolvedSteps) == 0 { + resolvedSteps = DefaultExtractionSteps + } + + resolved := &Config{ + Ignore: resolvedIgnore, + ExtractionSteps: resolvedSteps, + } + + // 1. Resolve Model ID: Default > YAML > Env > Flag + resolved.ModelID = OllamaDefaultModelID + if cfg.ModelID != "" { + resolved.ModelID = cfg.ModelID + } + if envVal := os.Getenv(CodeReducerModelIDEnvKey); envVal != "" { + resolved.ModelID = envVal + } + if modelIDFlag != "" { + resolved.ModelID = modelIDFlag + } + + // 2. Resolve Ollama Base URL: Default > YAML > Env + resolved.OllamaBaseURL = OllamaDefaultBaseURL + if cfg.OllamaBaseURL != "" { + resolved.OllamaBaseURL = cfg.OllamaBaseURL + } + if envVal := os.Getenv(OllamaBaseURLEnvKey); envVal != "" { + resolved.OllamaBaseURL = envVal + } + + // 3. Resolve Ollama Context Size: Default > YAML > Env > Flag + resolved.OllamaNumCtx = OllamaDefaultNumCtx + if cfg.OllamaNumCtx > 0 { + resolved.OllamaNumCtx = cfg.OllamaNumCtx + } + if envVal := os.Getenv(OllamaNumCtxEnvKey); envVal != "" { + if n, err := strconv.Atoi(envVal); err == nil && n > 0 { + resolved.OllamaNumCtx = n + } + } + if numCtxFlag != "" { + if n, err := strconv.Atoi(numCtxFlag); err == nil && n > 0 { + resolved.OllamaNumCtx = n + } + } + + // 4. Resolve DocsDir: Default > YAML + resolved.DocsDir = DefaultDocsDir + if cfg.DocsDir != "" { + resolved.DocsDir = cfg.DocsDir + } + + // 5. Resolve prompts: Config > Default + resolved.SystemPrompt = DefaultSystemPrompt + if cfg.SystemPrompt != "" { + resolved.SystemPrompt = cfg.SystemPrompt + } + resolved.ModuleSynthesisPrompt = DefaultModuleSynthesisPrompt + if cfg.ModuleSynthesisPrompt != "" { + resolved.ModuleSynthesisPrompt = cfg.ModuleSynthesisPrompt + } + resolved.ArchitecturePrompt = DefaultArchitecturePrompt + if cfg.ArchitecturePrompt != "" { + resolved.ArchitecturePrompt = cfg.ArchitecturePrompt + } + resolved.FileFactConsolidationPrompt = DefaultFileFactConsolidationPrompt + if cfg.FileFactConsolidationPrompt != "" { + resolved.FileFactConsolidationPrompt = cfg.FileFactConsolidationPrompt + } + + return resolved, nil +} diff --git a/internal/engine/cache.go b/internal/engine/cache.go index c18259d..9cd896c 100644 --- a/internal/engine/cache.go +++ b/internal/engine/cache.go @@ -9,6 +9,7 @@ import ( "os" "path/filepath" + "github.com/arrase/code-reducer/internal/config" "github.com/arrase/code-reducer/internal/tools" ) @@ -17,22 +18,34 @@ type FileCacheEntry struct { Facts string `json:"facts"` } +const currentCacheVersion = 1 + type MetadataCache struct { - Files map[string]FileCacheEntry `json:"files"` - Modules map[string]string `json:"modules"` + Version int `json:"version"` + StepsHash string `json:"steps_hash"` + Files map[string]FileCacheEntry `json:"files"` + Modules map[string]string `json:"modules"` +} + +func computeStepsHash(steps []config.ExtractionStep) string { + data, _ := json.Marshal(steps) + h := sha256.Sum256(data) + return hex.EncodeToString(h[:]) } func loadMetadataCache(repoRoot string, docsDir string) (*MetadataCache, error) { - metadataPath := filepath.Join(docsDir, ".metadata.json") + metadataPath := filepath.Join(docsDir, metadataFileName) data, err := tools.ReadFileSafely(repoRoot, metadataPath) if err != nil { if errors.Is(err, os.ErrNotExist) { return &MetadataCache{ + Version: currentCacheVersion, Files: make(map[string]FileCacheEntry), Modules: make(map[string]string), }, nil } return &MetadataCache{ + Version: currentCacheVersion, Files: make(map[string]FileCacheEntry), Modules: make(map[string]string), }, fmt.Errorf("failed to read metadata cache: %w", err) @@ -40,10 +53,19 @@ func loadMetadataCache(repoRoot string, docsDir string) (*MetadataCache, error) var cache MetadataCache if err := json.Unmarshal(data, &cache); err != nil { return &MetadataCache{ + Version: currentCacheVersion, Files: make(map[string]FileCacheEntry), Modules: make(map[string]string), }, fmt.Errorf("failed to unmarshal metadata cache: %w", err) } + if cache.Version != currentCacheVersion { + // Incompatible version: return a clean cache + return &MetadataCache{ + Version: currentCacheVersion, + Files: make(map[string]FileCacheEntry), + Modules: make(map[string]string), + }, nil + } if cache.Files == nil { cache.Files = make(map[string]FileCacheEntry) } @@ -55,13 +77,14 @@ func loadMetadataCache(repoRoot string, docsDir string) (*MetadataCache, error) // IsInitialized checks if the metadata cache file exists. func IsInitialized(repoRoot, docsDir string) bool { - metadataPath := filepath.Join(docsDir, ".metadata.json") + metadataPath := filepath.Join(docsDir, metadataFileName) _, err := tools.ReadFileSafely(repoRoot, metadataPath) return err == nil } func saveMetadataCache(repoRoot string, docsDir string, cache *MetadataCache) error { - metadataPath := filepath.Join(docsDir, ".metadata.json") + cache.Version = currentCacheVersion + metadataPath := filepath.Join(docsDir, metadataFileName) data, err := json.MarshalIndent(cache, "", " ") if err != nil { return err diff --git a/internal/engine/chunking.go b/internal/engine/chunking.go index 613b09f..bf22558 100644 --- a/internal/engine/chunking.go +++ b/internal/engine/chunking.go @@ -5,45 +5,65 @@ import ( "fmt" "strings" "unicode/utf8" + + "github.com/arrase/code-reducer/internal/config" ) -func reduceInChunks(ctx context.Context, c *LLMClient, nodePath string, items []string, logEvent func(string, string)) (string, error) { +func reduceWithLLM( + ctx context.Context, + c llmCaller, + items []string, + sysPrompt string, + buildPrompt func(batch []string) string, + logMsg func(batch []string) string, + errMsg string, + logEvent LogEventFunc, +) (string, error) { if len(items) == 0 { return "", nil } - maxChars := c.NumCtx * 3 - return reduceItems(items, maxChars, func(batch []string) (string, error) { - prompt := fmt.Sprintf("Synthesize architecture for %s:\n%s", nodePath, strings.Join(batch, "\n\n")) - logEvent("status", fmt.Sprintf("➜ LLM Synthesizing chunk for %s (%d items)", nodePath, len(batch))) - res, err := c.CallLLM(ctx, c.GetDefaultSystemPrompt("module_synthesis"), []Message{{Role: "user", Content: prompt}}, false) + maxChars := c.NumCtx() * maxCharsMultiplier + return reduceItems(ctx, items, maxChars, func(batch []string) (string, error) { + prompt := buildPrompt(batch) + logEvent(EventStatus, logMsg(batch)) + res, err := c.CallLLM(ctx, sysPrompt, []Message{{Role: "user", Content: prompt}}, false) if err != nil { - return "", fmt.Errorf("LLM error during synthesis: %w", err) + return "", fmt.Errorf("%s: %w", errMsg, err) } - return StripOuterMarkdownFence(res), nil + return stripOuterMarkdownFence(res), nil }) } -func reduceFileFacts(ctx context.Context, c *LLMClient, filePath string, stepName string, items []string, logEvent func(string, string)) (string, error) { - if len(items) == 0 { - return "", nil +func reduceInChunks(ctx context.Context, c llmCaller, nodePath string, items []string, cfg *config.Config, logEvent LogEventFunc) (string, error) { + sysPrompt := cfg.SystemPrompt + "\n" + cfg.ModuleSynthesisPrompt + buildPrompt := func(batch []string) string { + return fmt.Sprintf("Synthesize architecture for %s:\n%s", nodePath, strings.Join(batch, "\n\n")) + } + logMsg := func(batch []string) string { + return fmt.Sprintf("➜ LLM Synthesizing chunk for %s (%d items)", nodePath, len(batch)) } + return reduceWithLLM(ctx, c, items, sysPrompt, buildPrompt, logMsg, "LLM error during synthesis", logEvent) +} + +func reduceFileFacts(ctx context.Context, c llmCaller, filePath string, stepName string, items []string, cfg *config.Config, logEvent LogEventFunc) (string, error) { if len(items) == 1 { return items[0], nil } - maxChars := c.NumCtx * 3 - return reduceItems(items, maxChars, func(batch []string) (string, error) { - prompt := fmt.Sprintf("Consolidate and deduplicate the extracted facts for %s regarding step '%s':\n%s", filePath, stepName, strings.Join(batch, "\n\n")) - logEvent("status", fmt.Sprintf("➜ LLM Consolidating facts for %s (%d items)", filePath, len(batch))) - sysPrompt := c.GetBaseSystemPrompt() + "You are a specialized code documentation assistant. Consolidate, deduplicate and merge the following facts extracted from different chunks of the same file into a single, cohesive summary." - res, err := c.CallLLM(ctx, sysPrompt, []Message{{Role: "user", Content: prompt}}, false) - if err != nil { - return "", fmt.Errorf("LLM error during file fact consolidation: %w", err) - } - return StripOuterMarkdownFence(res), nil - }) + sysPrompt := cfg.SystemPrompt + "\n" + cfg.FileFactConsolidationPrompt + buildPrompt := func(batch []string) string { + return fmt.Sprintf("Consolidate and deduplicate the extracted facts for %s regarding step '%s':\n%s", filePath, stepName, strings.Join(batch, "\n\n")) + } + logMsg := func(batch []string) string { + return fmt.Sprintf("➜ LLM Consolidating facts for %s (%d items)", filePath, len(batch)) + } + return reduceWithLLM(ctx, c, items, sysPrompt, buildPrompt, logMsg, "LLM error during file fact consolidation", logEvent) } -func reduceItems(items []string, maxChars int, reduceFn func(batch []string) (string, error)) (string, error) { +func reduceItems(ctx context.Context, items []string, maxChars int, reduceFn func(batch []string) (string, error)) (string, error) { + if err := ctx.Err(); err != nil { + return "", err + } + // If there is exactly one item and it exceeds the max allowed character limit, truncate it. if len(items) == 1 && utf8.RuneCountInString(items[0]) > maxChars { runes := []rune(items[0]) @@ -55,13 +75,14 @@ func reduceItems(items []string, maxChars int, reduceFn func(batch []string) (st currentLen := 0 for _, item := range items { - if currentLen+len(item) > maxChars && len(currentBatch) > 0 { + itemRunes := utf8.RuneCountInString(item) + if currentLen+itemRunes > maxChars && len(currentBatch) > 0 { batches = append(batches, currentBatch) currentBatch = []string{item} - currentLen = len(item) + currentLen = itemRunes } else { currentBatch = append(currentBatch, item) - currentLen += len(item) + currentLen += itemRunes } } if len(currentBatch) > 0 { @@ -89,13 +110,16 @@ func reduceItems(items []string, maxChars int, reduceFn func(batch []string) (st var intermediate []string for _, batch := range batches { - chunkRes, err := reduceItems(batch, maxChars, reduceFn) + if err := ctx.Err(); err != nil { + return "", err + } + chunkRes, err := reduceItems(ctx, batch, maxChars, reduceFn) if err != nil { return "", err } intermediate = append(intermediate, chunkRes) } - return reduceItems(intermediate, maxChars, reduceFn) + return reduceItems(ctx, intermediate, maxChars, reduceFn) } // chunkTextWithOverlap splits text into chunks of maxRunes length, with overlapRunes of overlap between adjacent chunks. diff --git a/internal/engine/client.go b/internal/engine/client.go index 875f0c6..995660e 100644 --- a/internal/engine/client.go +++ b/internal/engine/client.go @@ -8,7 +8,6 @@ import ( "io" "net/http" "strings" - "time" ) type Message struct { @@ -16,22 +15,31 @@ type Message struct { Content string `json:"content"` } -type LLMClient struct { - ModelID string - BaseURL string - NumCtx int - HTTPClient *http.Client +type llmCaller interface { + CallLLM(ctx context.Context, systemPrompt string, messages []Message, jsonFormat bool) (string, error) + NumCtx() int } -func NewLLMClient(modelID, baseURL string, numCtx int) *LLMClient { - return &LLMClient{ - ModelID: modelID, - BaseURL: baseURL, - NumCtx: numCtx, - HTTPClient: &http.Client{Timeout: 10 * time.Minute}, +type llmClient struct { + modelID string + baseURL string + numCtx int + httpClient *http.Client +} + +func newLLMClient(modelID, baseURL string, numCtx int) *llmClient { + return &llmClient{ + modelID: modelID, + baseURL: baseURL, + numCtx: numCtx, + httpClient: &http.Client{Timeout: defaultHTTPTimeout}, } } +func (c *llmClient) NumCtx() int { + return c.numCtx +} + // Structs for Ollama requests and responses type ollamaRequest struct { Model string `json:"model"` @@ -49,15 +57,15 @@ type ollamaResponse struct { Message Message `json:"message"` } -// CallLLM invokes the LLM via HTTP failing fast without retries. -func (c *LLMClient) prepareOllamaRequest(ctx context.Context, systemPrompt string, messages []Message, stream bool, jsonFormat bool) (*http.Request, error) { - url := strings.TrimSuffix(c.BaseURL, "/") + "/api/chat" +// prepareOllamaRequest creates and serializes the HTTP request for the Ollama api/chat endpoint. +func (c *llmClient) prepareOllamaRequest(ctx context.Context, systemPrompt string, messages []Message, jsonFormat bool) (*http.Request, error) { + url := strings.TrimSuffix(c.baseURL, "/") + "/api/chat" reqBody := ollamaRequest{ - Model: c.ModelID, + Model: c.modelID, Messages: append([]Message{{Role: "system", Content: systemPrompt}}, messages...), - Stream: stream, - Options: &ollamaOptions{NumCtx: c.NumCtx}, + Stream: false, + Options: &ollamaOptions{NumCtx: c.numCtx}, } if jsonFormat { reqBody.Format = "json" @@ -78,14 +86,14 @@ func (c *LLMClient) prepareOllamaRequest(ctx context.Context, systemPrompt strin } // CallLLM invokes the LLM via HTTP failing fast without retries. -func (c *LLMClient) CallLLM(ctx context.Context, systemPrompt string, messages []Message, jsonFormat bool) (string, error) { +func (c *llmClient) CallLLM(ctx context.Context, systemPrompt string, messages []Message, jsonFormat bool) (string, error) { - req, err := c.prepareOllamaRequest(ctx, systemPrompt, messages, false, jsonFormat) + req, err := c.prepareOllamaRequest(ctx, systemPrompt, messages, jsonFormat) if err != nil { return "", err } - resp, err := c.HTTPClient.Do(req) + resp, err := c.httpClient.Do(req) if err != nil { return "", err } @@ -104,23 +112,8 @@ func (c *LLMClient) CallLLM(ctx context.Context, systemPrompt string, messages [ return result.Message.Content, nil } - body, _ := io.ReadAll(io.LimitReader(resp.Body, 1024)) + body, _ := io.ReadAll(io.LimitReader(resp.Body, maxErrorBodyBytes)) return "", fmt.Errorf("ollama api error: status %d, response: %s", resp.StatusCode, string(body)) } -func (c *LLMClient) GetBaseSystemPrompt() string { - return "You are Code-Reducer, an expert technical writer and code analyzer. Your job is to strictly follow instructions. You do not yap, you do not write filler.\n" + - "DEFENSIVE RULES: 1. Do NOT use absolute terms ('always', 'never', 'zero') unless explicitly proven. 2. Do NOT guess downstream consequences or invent unhandled paths. If an error is swallowed, just say it is swallowed. 3. Do NOT name standard library packages unless explicitly stated in the source text. 4. Only report facts you are 100% sure about.\n" -} -func (c *LLMClient) GetDefaultSystemPrompt(command string) string { - basePrompt := c.GetBaseSystemPrompt() - switch command { - case "module_synthesis": - return basePrompt + "Task: Write a technical documentation page for a code module based on the provided list of its internal components.\nRule 1: Group related functions and classes under appropriate Markdown headings.\nRule 2: Explain the responsibility of the module and the data flow.\nRule 3: Keep it highly technical and dense." - case "architecture": - return basePrompt + "Task: Write a global architecture or quickstart document based on the module summaries.\nRule 1: Explain the system boundaries and how the modules interact.\nRule 2: Provide a dense, developer-friendly overview." - default: - return basePrompt - } -} diff --git a/internal/engine/constants.go b/internal/engine/constants.go new file mode 100644 index 0000000..9264f4e --- /dev/null +++ b/internal/engine/constants.go @@ -0,0 +1,17 @@ +package engine + +import "time" + +const ( + defaultHTTPTimeout = 10 * time.Minute + maxErrorBodyBytes = 1024 + defaultChunkOverlap = 800 + minNumCtxFloor = 512 + contextWindowAllocRatio = 0.75 + maxCharsMultiplier = 3 + metadataFileName = ".metadata.json" + agentsFileName = "AGENTS.md" + defaultDirPerm = 0755 +) + +type LogEventFunc func(EventType, string) diff --git a/internal/engine/json_parser.go b/internal/engine/json_parser.go index 3279e35..3fbb585 100644 --- a/internal/engine/json_parser.go +++ b/internal/engine/json_parser.go @@ -1,106 +1,17 @@ package engine import ( - "encoding/json" - "fmt" "regexp" "strings" ) var markdownFenceRe = regexp.MustCompile("(?s)^\\x60{3,}(?:markdown|json)?\\s*(.*?)\\s*\\x60{3,}$") -// StripOuterMarkdownFence strips surrounding markdown or json code fences from input strings. -func StripOuterMarkdownFence(content string) string { +// stripOuterMarkdownFence strips surrounding markdown or json code fences from input strings. +func stripOuterMarkdownFence(content string) string { trimmed := strings.TrimSpace(content) if matches := markdownFenceRe.FindStringSubmatch(trimmed); len(matches) > 1 { return strings.TrimSpace(matches[1]) } return trimmed } - -func extractBalancedJSON(input string) (string, error) { - var stack []rune - inString := false - var escape bool - - for i, r := range input { - if escape { - escape = false - continue - } - if r == '\\' && inString { - escape = true - continue - } - if r == '"' { - inString = !inString - continue - } - - if !inString { - if r == '{' || r == '[' { - stack = append(stack, r) - } else if r == '}' || r == ']' { - if len(stack) > 0 { - last := stack[len(stack)-1] - if (r == '}' && last == '{') || (r == ']' && last == '[') { - stack = stack[:len(stack)-1] - } - } else { - return "", fmt.Errorf("unbalanced closing delimiter") - } - if len(stack) == 0 { - return input[:i+len(string(r))], nil - } - } - } - } - return "", fmt.Errorf("unbalanced delimiters") -} - -// CleanJSONResponse is kept for compatibility and wraps around extractBalancedJSON. -func CleanJSONResponse(response string) string { - trimmed := StripOuterMarkdownFence(response) - for i := 0; i < len(trimmed); i++ { - if trimmed[i] == '{' || trimmed[i] == '[' { - if candidate, err := extractBalancedJSON(trimmed[i:]); err == nil { - return candidate - } - } - } - return trimmed -} - -// UnmarshalJSONResponse cleans the raw response and unmarshals it into target. -func UnmarshalJSONResponse(rawResponse string, target interface{}) error { - trimmed := StripOuterMarkdownFence(rawResponse) - - // Collect all balanced JSON candidates starting with { or [ - var candidates []string - for i := 0; i < len(trimmed); i++ { - if trimmed[i] == '{' || trimmed[i] == '[' { - if candidate, err := extractBalancedJSON(trimmed[i:]); err == nil { - candidates = append(candidates, candidate) - } - } - } - - var lastErr error - for _, candidate := range candidates { - if err := json.Unmarshal([]byte(candidate), target); err == nil { - return nil - } else { - lastErr = err - } - } - - // Final fallback: try to unmarshal the whole trimmed string - if err := json.Unmarshal([]byte(trimmed), target); err == nil { - return nil - } - - if lastErr != nil { - return fmt.Errorf("failed to unmarshal JSON candidates: %w", lastErr) - } - return fmt.Errorf("no valid JSON found in response") -} diff --git a/internal/engine/orchestrator.go b/internal/engine/orchestrator.go index efc8f29..34c25f4 100644 --- a/internal/engine/orchestrator.go +++ b/internal/engine/orchestrator.go @@ -13,72 +13,85 @@ import ( "github.com/arrase/code-reducer/internal/tools" ) +// EventType defines the type of pipeline events. +type EventType string + +const ( + // EventStatus represents progress/status updates. + EventStatus EventType = "status" + // EventError represents execution errors. + EventError EventType = "error" +) + +// Event represents a progress or error notification from the pipeline. type Event struct { - Type string + Type EventType Message string } -func (c *LLMClient) GenerateStandardDocs(ctx context.Context, repoRoot, docsDir, rootSum string, logEvent func(string, string)) error { - logEvent("status", "Step 3: Global Architecture Synthesis...") +type orchestrator struct { + client llmCaller +} + +func (o *orchestrator) GenerateStandardDocs(ctx context.Context, repoRoot, docsDir, rootSum string, cfg *config.Config, logEvent LogEventFunc) error { + logEvent(EventStatus, "Step 3: Global Architecture Synthesis...") archPath := filepath.Join(docsDir, "architecture.md") archMsg := fmt.Sprintf("Write the global architecture overview (%s/architecture.md) based on the root summary.\n\n%s", docsDir, rootSum) - archDoc, err := c.CallLLM(ctx, c.GetDefaultSystemPrompt("architecture"), []Message{{Role: "user", Content: archMsg}}, false) + sysPrompt := cfg.SystemPrompt + "\n" + cfg.ArchitecturePrompt + archDoc, err := o.client.CallLLM(ctx, sysPrompt, []Message{{Role: "user", Content: archMsg}}, false) if err != nil { return fmt.Errorf("failed to generate global architecture: %w", err) } - if err := tools.WriteFileSafely(repoRoot, archPath, []byte(StripOuterMarkdownFence(archDoc))); err != nil { + if err := tools.WriteFileSafely(repoRoot, archPath, []byte(stripOuterMarkdownFence(archDoc))); err != nil { return fmt.Errorf("failed to write architecture.md: %w", err) } - logEvent("status", "Step 4: Generating Quickstart...") + logEvent(EventStatus, "Step 4: Generating Quickstart...") qsPath := filepath.Join(docsDir, "quickstart.md") qsMsg := fmt.Sprintf("Write the %s/quickstart.md page based on this architecture.\n\n%s", docsDir, rootSum) - qsDoc, err := c.CallLLM(ctx, c.GetDefaultSystemPrompt("architecture"), []Message{{Role: "user", Content: qsMsg}}, false) + qsDoc, err := o.client.CallLLM(ctx, sysPrompt, []Message{{Role: "user", Content: qsMsg}}, false) if err != nil { return fmt.Errorf("failed to generate quickstart documentation: %w", err) } - if err := tools.WriteFileSafely(repoRoot, qsPath, []byte(StripOuterMarkdownFence(qsDoc))); err != nil { + if err := tools.WriteFileSafely(repoRoot, qsPath, []byte(stripOuterMarkdownFence(qsDoc))); err != nil { return fmt.Errorf("failed to write quickstart.md: %w", err) } return nil } -func setupPipeline(repoRoot string, cfg *config.Config, logEvent func(string, string)) (string, []string, *MetadataCache) { +func setupPipeline(repoRoot string, cfg *config.Config, logEvent LogEventFunc) (string, []string, *MetadataCache) { docsDir := cfg.DocsDir gitignorePatterns, err := tools.LoadGitignore(repoRoot) if err != nil { - logEvent("status", fmt.Sprintf("Warning: failed to load .gitignore: %v", err)) + logEvent(EventStatus, fmt.Sprintf("Warning: failed to load .gitignore: %v", err)) } ignores := append(cfg.Ignore, docsDir) ignores = append(ignores, gitignorePatterns...) cache, err := loadMetadataCache(repoRoot, docsDir) if err != nil { - logEvent("status", fmt.Sprintf("Warning: failed to load metadata cache: %v", err)) + logEvent(EventStatus, fmt.Sprintf("Warning: failed to load metadata cache: %v", err)) } return docsDir, ignores, cache } -func teardownPipeline(repoRoot, docsDir string, cache *MetadataCache, logEvent func(string, string), successMsg string) { +func teardownPipeline(repoRoot, docsDir string, cache *MetadataCache, logEvent LogEventFunc, successMsg string) { if err := saveMetadataCache(repoRoot, docsDir, cache); err != nil { - logEvent("status", fmt.Sprintf("Warning: failed to save metadata cache: %v", err)) + logEvent(EventStatus, fmt.Sprintf("Warning: failed to save metadata cache: %v", err)) } - logEvent("status", successMsg) + logEvent(EventStatus, successMsg) } -func (c *LLMClient) RunInit(ctx context.Context, repoRoot string, cfg *config.Config, onEvent func(Event)) error { - logEvent := func(t, m string) { - if onEvent != nil { - onEvent(Event{Type: t, Message: m}) - } - } - logEvent("status", "Starting Map-Reduce pipeline: init") - logEvent("status", "Step 1: Code Discovery & Building Tree...") +func (o *orchestrator) RunInit(ctx context.Context, repoRoot string, cfg *config.Config, onEvent func(Event)) error { + logEvent := makeLogEvent(onEvent) + logEvent(EventStatus, "Starting Map-Reduce pipeline: init") + logEvent(EventStatus, "Step 1: Code Discovery & Building Tree...") docsDir, ignores, cache := setupPipeline(repoRoot, cfg, logEvent) + cache.StepsHash = computeStepsHash(cfg.ExtractionSteps) - codeFiles, err := tools.DiscoverCodeFiles(repoRoot, ignores, cfg.IgnoreExtensions) + codeFiles, err := tools.DiscoverCodeFiles(repoRoot, ignores) if err != nil { return err } @@ -88,16 +101,18 @@ func (c *LLMClient) RunInit(ctx context.Context, repoRoot string, cfg *config.Co hash, err := computeSHA256(repoRoot, f) if err == nil { precalculatedHashes[f] = hash + } else { + logEvent(EventStatus, fmt.Sprintf("Warning: failed to compute hash for %s: %v", f, err)) } } tree := buildTree(codeFiles) modulesDir := filepath.Join(repoRoot, docsDir, "modules") - if err := os.MkdirAll(modulesDir, 0755); err != nil { + if err := os.MkdirAll(modulesDir, defaultDirPerm); err != nil { return fmt.Errorf("failed to create modules directory: %w", err) } - logEvent("status", "Step 2: Hierarchical Tree-Merging (Map-Reduce)...") + logEvent(EventStatus, "Step 2: Hierarchical Tree-Merging (Map-Reduce)...") affectedDirs := make(map[string]bool) var markAllAffected func(n *DirNode) @@ -109,17 +124,28 @@ func (c *LLMClient) RunInit(ctx context.Context, repoRoot string, cfg *config.Co } markAllAffected(tree) - rootSum, err := synthesizeNode(ctx, c, tree, repoRoot, cfg, cache, affectedDirs, precalculatedHashes, logEvent) + pCtx := &pipelineContext{ + ctx: ctx, + client: o.client, + repoRoot: repoRoot, + cfg: cfg, + cache: cache, + affectedDirs: affectedDirs, + precalculatedHashes: precalculatedHashes, + logEvent: logEvent, + } + + rootSum, err := synthesizeNode(pCtx, tree) if err != nil { return err } - if err := c.GenerateStandardDocs(ctx, repoRoot, docsDir, rootSum, logEvent); err != nil { + if err := o.GenerateStandardDocs(ctx, repoRoot, docsDir, rootSum, cfg, logEvent); err != nil { return err } - logEvent("status", "Step 5: Updating AGENTS.md...") - agentFilePath := "AGENTS.md" + logEvent(EventStatus, fmt.Sprintf("Step 5: Updating %s...", agentsFileName)) + agentFilePath := agentsFileName agentGuidelines := fmt.Sprintf(`# AI Agent Guidelines This repository contains automatically generated documentation under the %s directory to help AI coding agents understand the system architecture, design patterns, and module structure: @@ -134,7 +160,7 @@ Before making changes, analyze these files to align with existing design choices agentFileBytes, err := tools.ReadFileSafely(repoRoot, agentFilePath) if err != nil { if err := tools.WriteFileSafely(repoRoot, agentFilePath, []byte(agentGuidelines)); err != nil { - return fmt.Errorf("failed to write AGENTS.md: %w", err) + return fmt.Errorf("failed to write %s: %w", agentsFileName, err) } } else { content := string(agentFileBytes) @@ -147,7 +173,7 @@ Before making changes, analyze these files to align with existing design choices } newContent := content + separator + agentGuidelines if err := tools.WriteFileSafely(repoRoot, agentFilePath, []byte(newContent)); err != nil { - return fmt.Errorf("failed to append to AGENTS.md: %w", err) + return fmt.Errorf("failed to append to %s: %w", agentsFileName, err) } } } @@ -156,19 +182,23 @@ Before making changes, analyze these files to align with existing design choices return nil } -func (c *LLMClient) RunUpdate(ctx context.Context, repoRoot string, cfg *config.Config, onEvent func(Event)) error { - logEvent := func(t, m string) { - if onEvent != nil { - onEvent(Event{Type: t, Message: m}) - } - } - logEvent("status", "Starting Map-Reduce pipeline: update") +func (o *orchestrator) RunUpdate(ctx context.Context, repoRoot string, cfg *config.Config, onEvent func(Event)) error { + logEvent := makeLogEvent(onEvent) + logEvent(EventStatus, "Starting Map-Reduce pipeline: update") docsDir, ignores, cache := setupPipeline(repoRoot, cfg, logEvent) - logEvent("status", "Step 1: Detecting changed files...") + currentStepsHash := computeStepsHash(cfg.ExtractionSteps) + if cache.StepsHash != currentStepsHash { + logEvent(EventStatus, "Warning: Extraction steps changed. Invalidating metadata cache to force full documentation regeneration.") + cache.Files = make(map[string]FileCacheEntry) + cache.Modules = make(map[string]string) + cache.StepsHash = currentStepsHash + } + + logEvent(EventStatus, "Step 1: Detecting changed files...") - codeFiles, err := tools.DiscoverCodeFiles(repoRoot, ignores, cfg.IgnoreExtensions) + codeFiles, err := tools.DiscoverCodeFiles(repoRoot, ignores) if err != nil { return err } @@ -178,11 +208,13 @@ func (c *LLMClient) RunUpdate(ctx context.Context, repoRoot string, cfg *config. var allowedCodeFiles []string for _, f := range codeFiles { - allowedCodeFiles = append(allowedCodeFiles, f) hash, err := computeSHA256(repoRoot, f) - if err == nil { - currentFilesMap[f] = hash + if err != nil { + logEvent(EventStatus, fmt.Sprintf("Warning: failed to compute hash for %s: %v", f, err)) + continue } + currentFilesMap[f] = hash + allowedCodeFiles = append(allowedCodeFiles, f) } for _, f := range allowedCodeFiles { @@ -233,7 +265,7 @@ func (c *LLMClient) RunUpdate(ctx context.Context, repoRoot string, cfg *config. for mPath := range cache.Modules { if !activeDirs[mPath] { delete(cache.Modules, mPath) - moduleFile := filepath.Join(docsDir, "modules", ToSafeMarkdownFilename(mPath)) + moduleFile := filepath.Join(docsDir, "modules", toSafeMarkdownFilename(mPath)) if absModuleFile, err := security.SafeResolve(repoRoot, moduleFile); err == nil { _ = os.Remove(absModuleFile) } @@ -244,12 +276,23 @@ func (c *LLMClient) RunUpdate(ctx context.Context, repoRoot string, cfg *config. propagateAffected(tree, affectedDirs) if len(affectedDirs) == 0 { - logEvent("status", "No modifications detected. Documentation is up to date.") + logEvent(EventStatus, "No modifications detected. Documentation is up to date.") return nil } - logEvent("status", fmt.Sprintf("Step 2: Hierarchical Tree-Merging (Map-Reduce)... (Affected modules: %d)", len(affectedDirs))) - rootSum, err := synthesizeNode(ctx, c, tree, repoRoot, cfg, cache, affectedDirs, currentFilesMap, logEvent) + logEvent(EventStatus, fmt.Sprintf("Step 2: Hierarchical Tree-Merging (Map-Reduce)... (Affected modules: %d)", len(affectedDirs))) + pCtx := &pipelineContext{ + ctx: ctx, + client: o.client, + repoRoot: repoRoot, + cfg: cfg, + cache: cache, + affectedDirs: affectedDirs, + precalculatedHashes: currentFilesMap, + logEvent: logEvent, + } + + rootSum, err := synthesizeNode(pCtx, tree) if err != nil { return err } @@ -270,11 +313,11 @@ func (c *LLMClient) RunUpdate(ctx context.Context, repoRoot string, cfg *config. } if affectedDirs["."] || !archExists || !qsExists { - if err := c.GenerateStandardDocs(ctx, repoRoot, docsDir, rootSum, logEvent); err != nil { + if err := o.GenerateStandardDocs(ctx, repoRoot, docsDir, rootSum, cfg, logEvent); err != nil { return err } } else { - logEvent("status", "Global architecture and quickstart pages are up to date. Skipping regeneration.") + logEvent(EventStatus, "Global architecture and quickstart pages are up to date. Skipping regeneration.") } teardownPipeline(repoRoot, docsDir, cache, logEvent, "Pipeline update completed successfully!") diff --git a/internal/engine/runner.go b/internal/engine/runner.go index 51b1e5a..cde188b 100644 --- a/internal/engine/runner.go +++ b/internal/engine/runner.go @@ -8,26 +8,35 @@ import ( "github.com/arrase/code-reducer/internal/security" ) +// Mode represents the operation mode of the documentation pipeline. +type Mode string + +const ( + // ModeInit initializes the project documentation. + ModeInit Mode = "init" + // ModeUpdate updates the existing project documentation incrementally. + ModeUpdate Mode = "update" +) + +// Runner orchestrates the execution of the documentation pipeline. type Runner struct { cfg *config.Config } +// NewRunner creates a new Runner instance with the given configuration. func NewRunner(cfg *config.Config) *Runner { return &Runner{ cfg: cfg, } } -func (r *Runner) Run(ctx context.Context, repoRoot string, mode string, onEvent func(Event)) error { - logEvent := func(t, m string) { - if onEvent != nil { - onEvent(Event{Type: t, Message: m}) - } - } +// Run executes the documentation pipeline for the specified mode. +func (r *Runner) Run(ctx context.Context, repoRoot string, mode Mode, onEvent func(Event)) error { + logEvent := makeLogEvent(onEvent) // 1. Ensure lockfile is in gitignore if err := security.EnsureGitignoreHasLockfile(repoRoot); err != nil { - logEvent("status", fmt.Sprintf("Warning: failed to ensure gitignore has lockfile: %v", err)) + logEvent(EventStatus, fmt.Sprintf("Warning: failed to ensure gitignore has lockfile: %v", err)) } // 2. Acquire repository lock @@ -37,16 +46,17 @@ func (r *Runner) Run(ctx context.Context, repoRoot string, mode string, onEvent } defer lock.Unlock() - // 3. Instantiate LLM Client - client := NewLLMClient(r.cfg.ModelID, r.cfg.OllamaBaseURL, r.cfg.OllamaNumCtx) + // 3. Instantiate LLM Client & Orchestrator + client := newLLMClient(r.cfg.ModelID, r.cfg.OllamaBaseURL, r.cfg.OllamaNumCtx) + orch := &orchestrator{client: client} // 4. Run the documentation pipeline - if mode == "init" { - if err := client.RunInit(ctx, repoRoot, r.cfg, onEvent); err != nil { + if mode == ModeInit { + if err := orch.RunInit(ctx, repoRoot, r.cfg, onEvent); err != nil { return err } - } else if mode == "update" { - if err := client.RunUpdate(ctx, repoRoot, r.cfg, onEvent); err != nil { + } else if mode == ModeUpdate { + if err := orch.RunUpdate(ctx, repoRoot, r.cfg, onEvent); err != nil { return err } } else { diff --git a/internal/engine/synthesize.go b/internal/engine/synthesize.go index 465a9bf..078cb6b 100644 --- a/internal/engine/synthesize.go +++ b/internal/engine/synthesize.go @@ -13,15 +13,26 @@ import ( "github.com/arrase/code-reducer/internal/tools" ) -func synthesizeNode(ctx context.Context, c *LLMClient, node *DirNode, repoRoot string, cfg *config.Config, cache *MetadataCache, affectedDirs map[string]bool, precalculatedHashes map[string]string, logEvent func(string, string)) (string, error) { - if err := ctx.Err(); err != nil { +type pipelineContext struct { + ctx context.Context + client llmCaller + repoRoot string + cfg *config.Config + cache *MetadataCache + affectedDirs map[string]bool + precalculatedHashes map[string]string + logEvent LogEventFunc +} + +func synthesizeNode(p *pipelineContext, node *DirNode) (string, error) { + if err := p.ctx.Err(); err != nil { return "", err } // If this node (and all descendants) is NOT affected, reuse cached summary! - if !affectedDirs[node.Path] && cache.Modules[node.Path] != "" { - logEvent("status", fmt.Sprintf("➜ Reusing cached summary for directory: %s", node.Path)) - return cache.Modules[node.Path], nil + if !p.affectedDirs[node.Path] && p.cache.Modules[node.Path] != "" { + p.logEvent(EventStatus, fmt.Sprintf("➜ Reusing cached summary for directory: %s", node.Path)) + return p.cache.Modules[node.Path], nil } var childNames []string @@ -33,7 +44,7 @@ func synthesizeNode(ctx context.Context, c *LLMClient, node *DirNode, repoRoot s childSummaries := make(map[string]string) for _, name := range childNames { child := node.Children[name] - sum, err := synthesizeNode(ctx, c, child, repoRoot, cfg, cache, affectedDirs, precalculatedHashes, logEvent) + sum, err := synthesizeNode(p, child) if err != nil { return "", err } @@ -47,21 +58,21 @@ func synthesizeNode(ctx context.Context, c *LLMClient, node *DirNode, repoRoot s // Calculate a 100% dynamic file truncation limit based purely on the context size. // Assuming ~4 characters per token, we allocate 75% of the total context window // for the file content, proportionally reserving the remaining 25% for prompts and output. - numCtx := c.NumCtx - if numCtx < 512 { - numCtx = 512 + numCtx := p.client.NumCtx() + if numCtx < minNumCtxFloor { + numCtx = minNumCtxFloor } - fileLimit := int(float64(numCtx * 4) * 0.75) + fileLimit := int(float64(numCtx * 4) * contextWindowAllocRatio) for _, f := range node.Files { - if err := ctx.Err(); err != nil { + if err := p.ctx.Err(); err != nil { return "", err } var fileHash string var ok bool - if precalculatedHashes != nil { - fileHash, ok = precalculatedHashes[f] + if p.precalculatedHashes != nil { + fileHash, ok = p.precalculatedHashes[f] } var contentBytes []byte @@ -69,14 +80,14 @@ func synthesizeNode(ctx context.Context, c *LLMClient, node *DirNode, repoRoot s // Check cache hit using precalculated hash without reading file var facts string - cachedEntry, exists := cache.Files[f] + cachedEntry, exists := p.cache.Files[f] if ok && exists && cachedEntry.SHA256 == fileHash { facts = cachedEntry.Facts } else { // Read file only on cache miss - contentBytes, err = tools.ReadFileSafely(repoRoot, f) + contentBytes, err = tools.ReadFileSafely(p.repoRoot, f) if err != nil { - logEvent("status", fmt.Sprintf("Warning: failed to read file %s: %v", f, err)) + p.logEvent(EventStatus, fmt.Sprintf("Warning: failed to read file %s: %v", f, err)) continue } @@ -92,33 +103,40 @@ func synthesizeNode(ctx context.Context, c *LLMClient, node *DirNode, repoRoot s } if facts == "" { + if contentBytes == nil { + contentBytes, err = tools.ReadFileSafely(p.repoRoot, f) + if err != nil { + p.logEvent(EventStatus, fmt.Sprintf("Warning: failed to read file %s: %v", f, err)) + continue + } + } contentStr := string(contentBytes) - overlap := 800 + overlap := defaultChunkOverlap if overlap > fileLimit/4 { overlap = fileLimit / 4 } chunks := chunkTextWithOverlap(contentStr, fileLimit, overlap) var factsBuilder strings.Builder - for i, step := range cfg.ExtractionSteps { + for i, step := range p.cfg.ExtractionSteps { var stepFacts []string for chunkIdx, chunk := range chunks { chunkMsg := "" if len(chunks) > 1 { chunkMsg = fmt.Sprintf(" (Chunk %d of %d)", chunkIdx+1, len(chunks)) } - logEvent("status", fmt.Sprintf("➜ Extracting file (Step %d/%d - %s)%s: %s", i+1, len(cfg.ExtractionSteps), step.Name, chunkMsg, f)) + p.logEvent(EventStatus, fmt.Sprintf("➜ Extracting file (Step %d/%d - %s)%s: %s", i+1, len(p.cfg.ExtractionSteps), step.Name, chunkMsg, f)) - systemPrompt := c.GetBaseSystemPrompt() + step.Prompt + systemPrompt := p.cfg.SystemPrompt + "\n" + step.Prompt userContent := fmt.Sprintf("File: %s%s inside Module: %s\n```\n%s\n```", filepath.Base(f), chunkMsg, node.Path, chunk) - res, err := c.CallLLM(ctx, systemPrompt, []Message{{Role: "user", Content: userContent}}, false) + res, err := p.client.CallLLM(p.ctx, systemPrompt, []Message{{Role: "user", Content: userContent}}, false) if err != nil { return "", fmt.Errorf("LLM error extracting %s for %s: %w", step.Name, f, err) } - stepFacts = append(stepFacts, StripOuterMarkdownFence(res)) + stepFacts = append(stepFacts, stripOuterMarkdownFence(res)) } - consolidatedFact, err := reduceFileFacts(ctx, c, f, step.Name, stepFacts, logEvent) + consolidatedFact, err := reduceFileFacts(p.ctx, p.client, f, step.Name, stepFacts, p.cfg, p.logEvent) if err != nil { return "", err } @@ -128,7 +146,7 @@ func synthesizeNode(ctx context.Context, c *LLMClient, node *DirNode, repoRoot s facts = strings.TrimSpace(factsBuilder.String()) // Update cache - cache.Files[f] = FileCacheEntry{ + p.cache.Files[f] = FileCacheEntry{ SHA256: fileHash, Facts: facts, } @@ -144,21 +162,21 @@ func synthesizeNode(ctx context.Context, c *LLMClient, node *DirNode, repoRoot s } if len(components) == 0 { - cache.Modules[node.Path] = "" + p.cache.Modules[node.Path] = "" return "", nil } - logEvent("status", fmt.Sprintf("➜ Synthesizing directory: %s (%d total components)", node.Path, len(components))) - finalSum, err := reduceInChunks(ctx, c, node.Path, components, logEvent) + p.logEvent(EventStatus, fmt.Sprintf("➜ Synthesizing directory: %s (%d total components)", node.Path, len(components))) + finalSum, err := reduceInChunks(p.ctx, p.client, node.Path, components, p.cfg, p.logEvent) if err != nil { return "", err } // Update module cache - cache.Modules[node.Path] = finalSum + p.cache.Modules[node.Path] = finalSum - modulePath := filepath.Join(cfg.DocsDir, "modules", ToSafeMarkdownFilename(node.Path)) - if err := tools.WriteFileSafely(repoRoot, modulePath, []byte(finalSum)); err != nil { + modulePath := filepath.Join(p.cfg.DocsDir, "modules", toSafeMarkdownFilename(node.Path)) + if err := tools.WriteFileSafely(p.repoRoot, modulePath, []byte(finalSum)); err != nil { return "", fmt.Errorf("failed to write module documentation for %s: %w", node.Path, err) } diff --git a/internal/engine/tree.go b/internal/engine/tree.go index ac5c3f2..23e355b 100644 --- a/internal/engine/tree.go +++ b/internal/engine/tree.go @@ -31,6 +31,9 @@ func determineAffected(node *DirNode, repoRoot, docsDir string, cache *MetadataC changedFiles := make(map[string]bool) for _, c := range filteredChanges { changedFiles[c.Path] = true + if c.Status == "Deleted" { + affectedDirs[path.Dir(c.Path)] = true + } } var checkNode func(n *DirNode) @@ -42,7 +45,7 @@ func determineAffected(node *DirNode, repoRoot, docsDir string, cache *MetadataC } } - safeName := ToSafeMarkdownFilename(n.Path) + safeName := toSafeMarkdownFilename(n.Path) modulePath := filepath.Join(docsDir, "modules", safeName) absModulePath, err := security.SafeResolve(repoRoot, modulePath) if err == nil { diff --git a/internal/engine/utils.go b/internal/engine/utils.go index cd80794..e03a133 100644 --- a/internal/engine/utils.go +++ b/internal/engine/utils.go @@ -4,11 +4,19 @@ import ( "strings" ) -// ToSafeMarkdownFilename converts a module path to a safe filename for markdown docs. -func ToSafeMarkdownFilename(modulePath string) string { +// toSafeMarkdownFilename converts a module path to a safe filename for markdown docs. +func toSafeMarkdownFilename(modulePath string) string { safeName := strings.ReplaceAll(modulePath, "/", "_") if safeName == "." || safeName == "" { safeName = "root" } return safeName + ".md" } + +func makeLogEvent(onEvent func(Event)) LogEventFunc { + return func(t EventType, m string) { + if onEvent != nil { + onEvent(Event{Type: t, Message: m}) + } + } +} diff --git a/internal/security/errors.go b/internal/security/errors.go new file mode 100644 index 0000000..601b39f --- /dev/null +++ b/internal/security/errors.go @@ -0,0 +1,10 @@ +package security + +import "errors" + +var ( + // ErrPathTraversal is returned when a path resolves outside the repository root. + ErrPathTraversal = errors.New("security violation: path traversal detected") + // ErrLockHeld is returned when another code-reducer process holds the file lock. + ErrLockHeld = errors.New("lock is already held by another process") +) diff --git a/internal/security/security.go b/internal/security/security.go index ac3471b..4c94d1a 100644 --- a/internal/security/security.go +++ b/internal/security/security.go @@ -8,7 +8,10 @@ import ( "sync" ) -const LockFileName = ".code-reducer.lock" +const ( + LockFileName = ".code-reducer.lock" + defaultFilePerm = 0644 +) // SafeResolve cleans the input path and ensures it lies strictly inside the repository. // It resolves symlinks on the existing ancestor parts to prevent path traversal via symlinks. @@ -59,7 +62,7 @@ func SafeResolve(repoRoot, inputPath string) (string, error) { // Verify that resolvedPath is inside resolvedRoot rel, err := filepath.Rel(resolvedRoot, resolvedPath) if err != nil || strings.HasPrefix(rel, "..") { - return "", fmt.Errorf("security violation: path traversal detected: %q", inputPath) + return "", fmt.Errorf("%w: %q", ErrPathTraversal, inputPath) } return resolvedPath, nil @@ -103,10 +106,10 @@ func AcquireLock(repoRoot string) (*SimpleLock, error) { return nil, err } - f, err := os.OpenFile(lockPath, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0644) + f, err := os.OpenFile(lockPath, os.O_WRONLY|os.O_CREATE|os.O_EXCL, defaultFilePerm) if err != nil { if os.IsExist(err) { - return nil, fmt.Errorf("lock at %s is already held by another process. If you are sure no other code-reducer process is running, delete this stale lockfile manually", lockPath) + return nil, fmt.Errorf("%w: lock at %s is already held by another process. If you are sure no other code-reducer process is running, delete this stale lockfile manually", ErrLockHeld, lockPath) } return nil, fmt.Errorf("failed to acquire lock at %s: %w", lockPath, err) } @@ -122,7 +125,10 @@ func AcquireLock(repoRoot string) (*SimpleLock, error) { // EnsureGitignoreHasLockfile ensures that the lockfile .code-reducer.lock is in the .gitignore. func EnsureGitignoreHasLockfile(repoRoot string) error { - gitignorePath := filepath.Join(repoRoot, ".gitignore") + gitignorePath, err := SafeResolve(repoRoot, ".gitignore") + if err != nil { + return err + } data, err := os.ReadFile(gitignorePath) if err != nil && !os.IsNotExist(err) { @@ -136,14 +142,39 @@ func EnsureGitignoreHasLockfile(repoRoot string) error { } } - f, err := os.OpenFile(gitignorePath, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644) + contentToAppend := "# Code-Reducer Lockfile\n" + LockFileName + "\n" + if len(data) > 0 && data[len(data)-1] != '\n' { + contentToAppend = "\n" + contentToAppend + } + + newData := append(data, []byte(contentToAppend)...) + + dir := filepath.Dir(gitignorePath) + tmpFile, err := os.CreateTemp(dir, ".gitignore.tmp.*") if err != nil { - return fmt.Errorf("failed to open .gitignore for appending: %w", err) + return fmt.Errorf("failed to create temp file for .gitignore: %w", err) + } + tmpName := tmpFile.Name() + defer func() { + tmpFile.Close() + os.Remove(tmpName) + }() + + if _, err := tmpFile.Write(newData); err != nil { + return fmt.Errorf("failed to write to temp file for .gitignore: %w", err) + } + if err := tmpFile.Sync(); err != nil { + return fmt.Errorf("failed to sync temp file for .gitignore: %w", err) + } + if err := tmpFile.Close(); err != nil { + return fmt.Errorf("failed to close temp file for .gitignore: %w", err) + } + if err := os.Chmod(tmpName, defaultFilePerm); err != nil { + return fmt.Errorf("failed to chmod temp file for .gitignore: %w", err) } - defer f.Close() - if _, err := f.WriteString("\n# Code-Reducer Lockfile\n" + LockFileName + "\n"); err != nil { - return fmt.Errorf("failed to write to .gitignore: %w", err) + if err := os.Rename(tmpName, gitignorePath); err != nil { + return fmt.Errorf("failed to rename temp file for .gitignore: %w", err) } return nil diff --git a/internal/tools/file_tools.go b/internal/tools/file_tools.go index daec3e7..9134be0 100644 --- a/internal/tools/file_tools.go +++ b/internal/tools/file_tools.go @@ -12,6 +12,12 @@ import ( ignore "github.com/sabhiram/go-gitignore" ) +const ( + binaryDetectionBufSize = 1024 + defaultDirPerm = 0755 + defaultFilePerm = 0644 +) + // ReadFileSafely resolves the virtual path inside the repository and reads the file content. // It implements TOCTOU mitigation similar to WriteFileSafely. func ReadFileSafely(repoRoot, virtualPath string) ([]byte, error) { @@ -61,45 +67,36 @@ func WriteFileSafely(repoRoot, virtualPath string, content []byte) error { } dir := filepath.Dir(safePath) - if err := os.MkdirAll(dir, 0755); err != nil { + if err := os.MkdirAll(dir, defaultDirPerm); err != nil { return fmt.Errorf("failed to create directory: %w", err) } - // Open the file without truncating first to prevent truncating a followed symlink target - f, err := os.OpenFile(safePath, os.O_WRONLY|os.O_CREATE, 0644) + tmpFile, err := os.CreateTemp(dir, filepath.Base(safePath)+".tmp.*") if err != nil { - return fmt.Errorf("failed to open file for writing: %w", err) + return fmt.Errorf("failed to create temp file: %w", err) } - defer f.Close() + tmpName := tmpFile.Name() + defer func() { + tmpFile.Close() + os.Remove(tmpName) + }() - // TOCTOU mitigation: verify the file path is not a symlink - fiLstat, err := os.Lstat(safePath) - if err != nil { - return fmt.Errorf("failed to lstat file: %w", err) + if _, err := tmpFile.Write(content); err != nil { + return fmt.Errorf("failed to write to temp file: %w", err) } - if fiLstat.Mode()&os.ModeSymlink != 0 { - return fmt.Errorf("security violation: symlink detected on write: %s", safePath) + if err := tmpFile.Sync(); err != nil { + return fmt.Errorf("failed to sync temp file: %w", err) } - - fiFstat, err := f.Stat() - if err != nil { - return fmt.Errorf("failed to fstat file: %w", err) - } - - // Verify the file descriptor points to the exact same file path check - if !os.SameFile(fiLstat, fiFstat) { - return fmt.Errorf("security violation: TOCTOU symlink race detected on write: %s", safePath) + if err := tmpFile.Close(); err != nil { + return fmt.Errorf("failed to close temp file: %w", err) } - - // Truncate safely only after verifying it is not a symlink - if err := f.Truncate(0); err != nil { - return fmt.Errorf("failed to truncate file: %w", err) + if err := os.Chmod(tmpName, defaultFilePerm); err != nil { + return fmt.Errorf("failed to chmod temp file: %w", err) } - if _, err := f.Write(content); err != nil { - return fmt.Errorf("failed to write file content: %w", err) + if err := os.Rename(tmpName, safePath); err != nil { + return fmt.Errorf("failed to rename temp file: %w", err) } - return nil } @@ -128,7 +125,7 @@ func LoadGitignore(repoRoot string) ([]string, error) { } // ShouldIgnoreFile checks if a file (specified by relative path) is ignored. -func ShouldIgnoreFile(repoRoot, relPath string, gitIgnore *ignore.GitIgnore, ignoredExtensions []string) bool { +func ShouldIgnoreFile(repoRoot, relPath string, gitIgnore *ignore.GitIgnore) bool { slashRelPath := filepath.ToSlash(relPath) // 1. Check user-defined ignores (config + gitignore) @@ -144,23 +141,9 @@ func ShouldIgnoreFile(repoRoot, relPath string, gitIgnore *ignore.GitIgnore, ign } } - // 3. Check filename extensions & suffixes + // 3. Check if it's a known text file to avoid IsBinaryFile I/O bottleneck name := filepath.Base(slashRelPath) ext := strings.ToLower(filepath.Ext(name)) - for _, iext := range ignoredExtensions { - lowerIext := strings.ToLower(iext) - lowerName := strings.ToLower(name) - if !strings.Contains(lowerIext, ".") { - if ext == "."+lowerIext || strings.HasSuffix(lowerName, "."+lowerIext) { - return true - } - } else { - if strings.HasSuffix(lowerName, lowerIext) { - return true - } - } - } - // 3.5 Check if it's a known text file to avoid IsBinaryFile I/O bottleneck knownTextExts := map[string]bool{ ".go": true, ".js": true, ".ts": true, ".py": true, ".md": true, ".txt": true, ".json": true, ".yaml": true, ".yml": true, @@ -183,12 +166,13 @@ func ShouldIgnoreFile(repoRoot, relPath string, gitIgnore *ignore.GitIgnore, ign // DiscoverCodeFiles recursively walks the codebase to find high-signal source files. // It ignores build, dependency, and output files, as well as any paths in the custom ignores list. -func DiscoverCodeFiles(repoRoot string, ignores []string, ignoredExtensions []string) ([]string, error) { +func DiscoverCodeFiles(repoRoot string, ignores []string) ([]string, error) { var files []string gitIgnore := ignore.CompileIgnoreLines(ignores...) err := filepath.WalkDir(repoRoot, func(path string, d os.DirEntry, err error) error { if err != nil { + fmt.Fprintf(os.Stderr, "Warning: error walking path %s: %v\n", path, err) return nil // Skip items with errors } @@ -198,6 +182,7 @@ func DiscoverCodeFiles(repoRoot string, ignores []string, ignoredExtensions []st rel, err := filepath.Rel(repoRoot, path) if err != nil { + fmt.Fprintf(os.Stderr, "Warning: failed to get relative path for %s: %v\n", path, err) return nil } @@ -211,7 +196,7 @@ func DiscoverCodeFiles(repoRoot string, ignores []string, ignoredExtensions []st return nil } - if ShouldIgnoreFile(repoRoot, slashRel, gitIgnore, ignoredExtensions) { + if ShouldIgnoreFile(repoRoot, slashRel, gitIgnore) { return nil } @@ -228,14 +213,14 @@ func DiscoverCodeFiles(repoRoot string, ignores []string, ignoredExtensions []st func IsBinaryFile(path string) bool { f, err := os.Open(path) if err != nil { - return false + return true } defer f.Close() - buf := make([]byte, 1024) + buf := make([]byte, binaryDetectionBufSize) n, err := f.Read(buf) if err != nil && err != io.EOF { - return false + return true } for i := 0; i < n; i++ { diff --git a/wiki/.metadata.json b/wiki/.metadata.json deleted file mode 100644 index b9b9d58..0000000 --- a/wiki/.metadata.json +++ /dev/null @@ -1,89 +0,0 @@ -{ - "files": { - "cmd/init.go": { - "sha256": "87cb2d0246334e1b067c69acf67d42eddd7bae533c7cea9b0cf156ef69a3fcc2", - "facts": "#### [API_SIGNATURES]\n## Public Surface Area of `cmd/init.go`\n\n**Exported items:** None.\n\n#### [BUSINESS_LOGIC]\n**Domain Concept:** CLI-driven documentation scaffolding tool (wiki generator).\n\n**Business Rule:** When the user runs `init`, the system scans the repository and generates an initial set of wiki markdown pages. This is a one-time setup operation for project documentation.\n\n**Algorithmic Flow:**\n\n1. User invokes the `init` subcommand on the CLI.\n2. The command delegates to `executeCommand(\"init\")`.\n3. That function (defined elsewhere in this module) performs the repository scan and wiki page generation.\n\n**Note:** All substantive logic is deferred to `executeCommand`, which is not included in this file. This file serves only as registration/wiring for the subcommand within the cobra command hierarchy.\n\n#### [STATE_AND_CONCURRENCY]\n**Mutable State:** None. The only variable (`initCmd`) is initialized once at package scope and never reassigned. No concurrent access or modification occurs.\n\n**Concurrency Mechanisms:** None present in this file.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects \u0026 I/O Analysis\n\n**No direct external communication.** This file contains zero network calls, disk operations, database queries, or other I/O primitives. All side effects are delegated to an out-of-scope function (`executeCommand`).\n\n**Delegated behavior:** The actual work is performed by `executeCommand(\"init\")`, which resolves at runtime via a reference not defined in this file. Side effects (file writes, HTTP calls, etc.) live in that external implementation.\n\n## Error Handling Analysis\n\n| Pattern | Used? |\n|---|---|\n| Wrap / aggregate errors | ❌ |\n| Sentinel error | ❌ |\n| Panic | ❌ |\n| Pass-through delegation | βœ… |\n\nThe `RunE` callback returns the raw result of `executeCommand(\"init\")` with no transformation. Whatever error (or nil) comes back from that function is propagated directly to cobra's command framework, which then surfaces it to the CLI user." - }, - "cmd/root.go": { - "sha256": "2a56f97199f6fad19a3f550934589865f0e8f48e3a46ab42a34a224a5e23065b", - "facts": "#### [API_SIGNATURES]\n## Public Surface Area\n\n### Exported Variables\n\n| Name | Type | Description |\n|------|------|-------------|\n| `RootCmd` | `*cobra.Command` | The root cobra command for the CLI. Initialized with Use=\"code-reducer\", Short, Long descriptions and CompletionOptions (DisableDefaultCmd=true). Has two PersistentFlags registered: `--model-id` and `--num-ctx`. |\n\n### Not Exported (Internal)\n\n| Name | Type | Description |\n|------|------|-------------|\n| `init()` | function | Registers persistent flags on RootCmd. |\n| `executeCommand(mode string)` | error | Internal command executor; not part of public API. |\n\n#### [BUSINESS_LOGIC]\n## Business Rules \u0026 Domain Concepts\n\n**Domain**: Code-Reducer is a documentation-generation agent. It writes and maintains project wikis by calling an LLM to produce markdown docs from source code.\n\n**Core Flow** (high-level algorithm):\n\n1. **Environment validation** β€” confirm the current working directory is a Git repository before proceeding.\n2. **Configuration resolution** β€” merge three sources: CLI flags (`--model-id`, `--num-ctx`), environment defaults, and an optional persistent config file. The resolved config drives all subsequent behavior.\n3. **Implicit setup on first run** β€” if the persistent config file does not exist yet and stdin is a terminal, trigger an automatic initial configuration step without requiring user intervention. On non-TTY (CI / piping), this step is skipped and the caller must configure manually.\n4. **Mode-gated lifecycle checks**:\n - `init` mode β†’ project must NOT already be initialized; otherwise error out.\n - `update` mode β†’ project MUST already be initialized; otherwise error out.\n These two modes are mutually exclusive at initialization time.\n5. **Graceful shutdown** β€” install SIGINT / SIGTERM handlers so the process exits cleanly when interrupted, rather than being killed mid-run.\n6. **Execution loop** β€” delegate to `engine.NewRunner.Run()` with the resolved config and current directory. The runner emits typed events (`status`, `error`, default). Each event is surfaced as a line of output (stderr for errors, stdout otherwise).\n\n**Key constraints**:\n- Requires Git repo at invocation time.\n- First-run auto-configures only when stdin is interactive.\n- `init` and `update` are mutually exclusive on an already-initialized project; you must run `code-reducer setup` first before either CLI command can work.\n\n#### [STATE_AND_CONCURRENCY]\n## Root.go - State Mutation \u0026 Concurrency Analysis\n\n### Mutable State\n\n| Variable | Scope | Mutability | Notes |\n|---|---|---|---|\n| `modelIdFlag` | Package-level (`var`) | Mutated once via `StringVar` in `init()` | Set by flag binding; value consumed in `executeCommand()` without further writes. |\n| `numCtxFlag` | Package-level (`var`) | Mutated once via `StringVar` in `init()` | Same pattern as above. |\n\nNo other fields or structs are mutated within this file after their initial assignment.\n\n### Concurrency Mechanisms\n\n**None.** No mutexes, channels, atomic operations, or other concurrency primitives appear in this file. The signal notification context (`signal.NotifyContext`) is used for cancellation but does not protect shared mutable stateβ€”each `executeCommand` invocation runs sequentially and consumes its own local variables.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects (I/O with the Outside World)\n\n| Operation | Direction | Mechanism | Notes |\n|---|---|---|---|\n| `os.Getwd()` | Read OS state | Filesystem call to resolve current working directory | Wrapped in error-return; caller handles failure. |\n| `isatty.IsTerminal(os.Stdin.Fd())` / `isatty.IsCygwinTerminal(os.Stdin.Fd())` | Read OS state | Stdio file descriptor inspection via third-party library (`go-isatty`) | Used only for the guard check before setup flow; result is not persisted. |\n| `fmt.Println(ev.Message)` (2 occurrences) | Write to stdout | Standard library `fmt` package | Triggered inside the callback passed to `runner.Run()` when event type is `\"status\"` or default case. |\n| `fmt.Fprintf(os.Stderr, \"Error: %s\\n\", ev.Message)` | Write to stderr | Standard library `fmt` package | Triggered inside the callback when event type is `\"error\"`. |\n\n**No network I/O is performed in this file.** Disk/DB access (config reads, git verification) is delegated to imported packages (`internal/config`, `internal/engine`, `internal/tools`) and their behavior is not observable from this file alone.\n\n---\n\n## Error Handling Strategy\n\n| Pattern | Evidence |\n|---|---|\n| **Error wrapping with `%w`** | Used on `os.Getwd()` failure: `fmt.Errorf(\"failed to get current working directory: %w\", err)`. Preserves chain traceability for callers that may want to unwrap. |\n| **Early-return propagation** | Every error path in `executeCommand()` returns immediately (e.g., git verification, config resolution, init/update state checks). No swallowed errors β€” each is propagated upward. |\n| **Explicit sentinel messages** | Two hardcoded messages returned as wrapped errors: `\"the project has already been initialized...\"` and `\"the project has not been initialized yet...\"`. These are user-facing error strings, not internal sentinels. |\n| **Final runner failure wrapping** | `runner.Run()`'s error is wrapped once: `fmt.Errorf(\"documentation run failed: %w\", err)`, then returned from the function. No further recovery or retry logic here. |\n| **Silenced Cobra output** | `RootCmd` has `SilenceErrors: true` and `SilenceUsage: true`. Errors that would otherwise be printed by cobra's default handler are suppressed β€” callers must handle errors explicitly (which this function does). |\n| **No panic() calls** | None observed. All failure paths return errors; the program exits via error propagation to cobra, not via process termination signals from within Go code. |\n\n---\n\n## Notable Observations\n\n- The `signal.NotifyContext` setup (`ctx, stop := signal.NotifyContext(...)`) installs OS-level interrupt handlers for `SIGINT` and `SIGTERM`. This is an I/O side effect (process signal handling) but has no direct error-return path β€” the deferred `stop()` cancels context; any in-flight work that respects the cancelled context will exit.\n- The callback passed to `runner.Run()` does not capture errors from event types `\"status\"` or default β€” it only prints them to stdout/stderr. Error events are routed to stderr, but there is no distinction between a \"real\" error and an informational message at this level; both use the same callback structure." - }, - "cmd/setup.go": { - "sha256": "b2933daf633b5b3529e0dfce295be7e5753502653a3661b445312eaa389e5a88", - "facts": "#### [API_SIGNATURES]\n# Public Surface Area: `cmd/setup.go`\n\n## Exported Functions\n\n| Function | Signature | Notes |\n|---|---|---|\n| `RunSetupFlow` | `(repoRoot string) error` | Interactive setup flow that generates a `.code-reducer.yaml` configuration file. |\n| `promptStringList` | `(reader *bufio.Reader, promptMsg string, existingList []string) []string` | Reads comma-separated list input from stdin. Returns existing values on I/O error or empty/`clear`/`none` input. |\n\n## Internal (Unexported) Functions\n\n| Function | Signature | Notes |\n|---|---|---|\n| `init` | `()` | Registers `setupCmd` with `RootCmd`. |\n| `promptString` | `(reader *bufio.Reader, promptMsg string, existingVal string) string` | Reads single-line text input from stdin. Returns existing value on I/O error or empty input. |\n\n## Internal State\n\n| Identifier | Type | Notes |\n|---|---|---|\n| `setupCmd` | `\\*cobra.Command` | The `setup` sub-command registered with the root cobra command. |\n\n#### [BUSINESS_LOGIC]\n## Business Logic \u0026 Domain Concepts\n\nThis file implements an **interactive configuration setup flow** for a CLI tool called Code-Reducer. The domain concept is **user-driven initial and reconfiguration of application settings**, persisted to disk.\n\n### High-Level Algorithmic Flow\n\n1. **Resolve existing state**: Attempt to load any previously saved config from the current working directory. If none exists, all prompts default to built-in constants (Ollama defaults for model ID, base URL, context size; `\"wiki\"` for docs directory).\n\n2. **Prompt user iteratively** for each setting:\n - LLM Model ID β†’ string\n - Ollama Base URL β†’ string\n - Ollama Context Size β†’ integer (parsed from the current value formatted as a string)\n - Custom directories/files to ignore β†’ comma-separated list\n - File extensions to ignore β†’ comma-separated list\n - Documentation directory β†’ string\n\n3. **Preserve existing values on empty input**: If the user presses Enter without typing anything, the previously loaded (or default) value is kept.\n\n4. **Support reset semantics**: For list-type inputs, the literal strings `\"clear\"` or `\"none\"` wipe the list entirely; otherwise comma-splitting produces the new list.\n\n5. **Error swallowing on read failures**: If stdin reading fails during any prompt, the error is logged to stderr and execution continues using existing values (no rollback, no restart).\n\n6. **Persist final config**: After all prompts complete, a single `config.Config` struct is constructed from user input + preserved state and written to disk via `config.SaveConfig`. The success message references a `.yaml` file path defined in the config package.\n\n### Key Business Rules\n\n- **Non-destructive defaults**: Existing configuration is always preferred over built-in constants; new values only replace what the user explicitly provided (or left as default).\n- **Idempotent re-run**: Running `setup` again when a config already exists produces an incremental update rather than a full overwrite.\n- **Graceful degradation on I/O errors**: Any stdin read failure does not abort setup; it silently falls back to existing state.\n\n#### [STATE_AND_CONCURRENCY]\n## Mutable State Analysis\n\n**Global Variables:** None are mutated after initialization in this file.\n\n**Local Mutable State (within `RunSetupFlow`):** Local function-scope variables (`existingModel`, `existingBaseURL`, `existingNumCtx`, `modelInput`, `urlInput`, etc.) β€” these are reassigned during the setup flow but are not shared across goroutines or protected by any concurrency mechanism.\n\n**Struct Fields:** The `config.Config` struct is constructed with new values and assigned fields sequentially, then saved. No concurrent access occurs.\n\n## Concurrency Mechanisms\n\n**None.** There are no channels, mutexes (`sync.Mutex`, `sync.RWMutex`), atomic types, or other synchronization primitives used in this file.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects \u0026 Error Handling Analysis\n\n### External Communication (I/O)\n\n**1. Disk reads (filesystem)**\n- `os.Getwd()` β€” reads current working directory from filesystem. Returns error if unavailable; error is wrapped and propagated to caller.\n- `config.LoadConfig(repoRoot)` β€” reads a YAML config file from disk at `repoRoot`. If this fails, `existingCfg` remains `nil` (no panic).\n\n**2. Disk writes (filesystem)**\n- `config.SaveConfig(repoRoot, newCfg)` β€” writes the new configuration to disk. On error, wraps and returns: `\"error saving configuration: %w\"`. No retry or fallback.\n\n**3. Stdin reads (user input)**\n- `bufio.NewReader(os.Stdin)` + `reader.ReadString('\\n')` β€” called 6 times total across `promptString` and `promptStringList`:\n - Model ID prompt\n - Base URL prompt\n - Context size prompt\n - Custom directories/files to ignore\n - Custom file extensions to ignore\n - Documentation directory\n\n**4. Stdout writes**\n- Welcome banner (`fmt.Println`)\n- Each prompt label (e.g., `\"Enter LLM Model ID [existing]:\"`)\n- Success confirmation: `Configuration successfully saved to local \u003cConfigFileName\u003e file.`\n\n**5. Stderr writes**\n- On every `ReadString` failure, prints warning to stderr with the error value and falls back to existing/default values. No log level; just a single-line notice per call site.\n\n### Error Handling Patterns\n\n| Failure point | Behavior |\n|---|---|\n| `os.Getwd()` | Wrapped into `fmt.Errorf(\"failed to get current working directory: %w\", err)` and returned up the stack. |\n| `config.LoadConfig(repoRoot)` | Nil-check only; if error occurs, existing values default to `OllamaDefault*` constants. No wrapping, no logging, no retry. Effectively swallowed. |\n| `strconv.Atoi(ctxInputStr)` | Nil-check only; on parse failure or non-positive result, falls back to `existingNumCtx`. Swallowed silently. |\n| `reader.ReadString('\\n')` in prompt functions | Written to stderr as `\"Warning: error reading input (%v), \u003cfallback\u003e\"`. Returns existing value (or empty for list). No propagation β€” caller never sees the error. |\n| `config.SaveConfig(repoRoot, newCfg)` | Wrapped with `%w`. Returned up the stack via `RunE` on the cobra command, so cobra will print it to stderr and exit non-zero. This is the only path that surfaces errors to the user without extra wrapping. |\n\n### Notable Observations\n\n- **No panic anywhere** β€” all I/O failures are handled (silently or with a warning).\n- **Two error strategies**: some paths *propagate* (`Getwd`, `SaveConfig`), others *swallow and fallback* (`LoadConfig`, `Atoi`, prompt readers). This inconsistency is intentional: setup tolerates missing config, but treats save failures as fatal.\n- **No retry logic** β€” `SaveConfig` is called once; if it fails, the entire flow aborts.\n- **Stdin errors are not logged to a structured logger** β€” they go directly to stderr with a single `fmt.Fprintf`." - }, - "cmd/update.go": { - "sha256": "62e9aa11268d32732195d0dbf3d38f5f817afe9a2a276aa7a1effac06566f8ea", - "facts": "#### [API_SIGNATURES]\n## Public Surface Area of `cmd/update.go`\n\nThis file exports **no public functions**, **no structs**, and **no interfaces**.\n\n---\n\n### Unexported Symbols (Internal Only)\n\n| Symbol | Kind | Notes |\n|--------|------|-------|\n| `updateCmd` | `*cobra.Command` | Local variable, not exported |\n| `init()` | function | Lowercase; registered via Go's `init()` convention |\n\n### Undeclared Reference\n\n- `executeCommand(string) error` β€” referenced but **not defined** in this file. Cannot be attributed to this module without further source inspection.\n\n#### [BUSINESS_LOGIC]\n**Business Domain Concept:** Project Documentation Management via CLI\n\n**Business Rule (stated):** When a project documentation system is updated incrementallyβ€”by scanning changed files since the last documented commitβ€”the wiki pages should be regenerated accordingly.\n\n**High-Level Algorithmic Flow:**\n1. The user invokes an `update` subcommand on the root CLI (`RootCmd`).\n2. Cobra parses and validates the command invocation.\n3. The handler delegates to `executeCommand(\"update\")`, which resolves the actual update logic (implementation not present in this file).\n\n**Notes:** This file only registers the CLI entry point; no business logic, validation, or data processing is contained here.\n\n#### [STATE_AND_CONCURRENCY]\n**No mutable state.** This file contains only command registration logic with immutable package-level variables (`updateCmd` is assigned once at init). No concurrent access control mechanisms (channels, mutexes) are present.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n# Analysis: Side Effects \u0026 Error Handling\n\n## External Communication (I/O)\n\n**None observed in this file.** The `update.go` contains no direct interactions with network, disk, database, or any external system. All I/O is delegated to an external function (`executeCommand`) whose implementation is not present in this file.\n\nThe cobra package import is a CLI framework dependency; it does not perform side effects itselfβ€”it only registers the command definition and delegates execution.\n\n## Error Handling\n\n**Error propagation via `RunE` return value.** The `updateCmd` handler returns errors from `executeCommand(\"update\")` directly without wrapping, catching, or transforming them:\n\n```go\nRunE: func(cmd *cobra.Command, args []string) error {\n return executeCommand(\"update\") // raw pass-through\n},\n```\n\n- **No error wrapping** β€” the returned error from `executeCommand` is surfaced to cobra as-is.\n- **No sentinel or custom errors** created in this file.\n- **No panic recovery.** If `executeCommand` panics, it propagates up unhandled (standard cobra behavior).\n\n## Summary\n\n| Concern | Status |\n|---|---|\n| I/O (network/disk/DB) | Not present; delegated to external function |\n| Error wrapping | None β€” raw pass-through |\n| Sentinel errors | None created here |\n| Panic handling | Absent |" - }, - "go.mod": { - "sha256": "360e06dd1314ef70f66d1f9b1c0c4c87ed7e1adbcf3e0c7e042a2056077d894b", - "facts": "#### [API_SIGNATURES]\nNo public surface area found. This `go.mod` file contains only module metadata (module path, Go version) and dependency declarations β€” no exported structs, interfaces, or methods are defined here.\n\n#### [BUSINESS_LOGIC]\n## Analysis: `go.mod`\n\n**What this file does:** None of the business logic or domain concepts you're looking for exist here. This is a Go module manifest β€” it declares dependencies, nothing more.\n\n### What's actually in this file:\n\n| Section | Purpose |\n|---------|---------|\n| `module github.com/arrase/code-reducer` | Identifies the project as belonging to the `code-reducer` package under user `arrase`. |\n| `go 1.26` | Requires Go 1.26 or newer to build this module. |\n| Direct dependencies | `go-isatty`, `go-gitignore`, `cobra`, `yaml.v3` β€” these are third-party libraries the project uses at runtime. |\n| Indirect dependencies | Transitive pulls from those libraries (e.g., `mousetrap`, `pflag`, `sys`). |\n\n### Dependencies hint at possible downstream behavior:\n\n- **`cobra`** β†’ CLI command parsing; suggests this is a terminal tool with subcommands.\n- **`go-isatty`** β†’ Detects whether stdin is a terminal (TTY). Used to conditionally render progress bars or prompts only when attached to a TTY.\n- **`go-gitignore`** β†’ Supports `.gitignore` style ignore patterns; likely used by the tool itself to skip files during processing.\n- **`yaml.v3`** β†’ Configures YAML parsing, suggesting configuration is loaded from YAML files at runtime.\n\n### Algorithmic flow: Not applicable.\n\nThere is no executable logic in `go.mod`. The actual behavior lives in `.go` source files elsewhere in the module β€” this file only tells Go's toolchain where to find packages.\n\n#### [STATE_AND_CONCURRENCY]\nNo mutable state.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects \u0026 Error Handling Analysis: `go.mod`\n\n### Communication with Outside World (I/O)\n\n**None.** This file is purely declarative metadata. It does not perform any runtime operations. The only interaction it has with the outside world occurs at module resolution time when the Go toolchain reads this file to determine dependencies β€” but that is a build-system concern, not application behavior.\n\n### Error Handling / Return Patterns\n\n**None.** There is no error handling in this file:\n- No panic/recover, no sentinel errors, no wrapped errors (`fmt.Errorf`, `errors.Join`, etc.)\n- No I/O operations (no network calls, no disk reads/writes beyond toolchain consumption)\n- No logging or structured output\n\n### Dependency Declarations (Side Effect Surface)\n\nThese are not runtime side effects β€” they are compile-time declarations. However, the *presence* of these dependencies constrains what the application can do:\n\n| Package | Implied Capability | Error Handling in That Context |\n|---|---|---|\n| `spf13/cobra` v1.10.2 | CLI argument parsing, subcommand dispatch | Cobra's own error handling (not visible here) |\n| `gopkg.in/yaml.v3` v3.0.1 | YAML decoding/encoding (`*yaml.Node`, `Unmarshal`) | Returns errors for bad input; caller must handle |\n| `mattn/go-isatty` v0.0.22 | Terminal detection via syscall | Standard library-level, no application-layer handling |\n| `sabhiram/go-gitignore` v0.0.0-... | `.gitignore` parsing | Returns errors for malformed patterns; caller must handle |\n\n### Indirect Dependencies\n\nThese are transitive and not directly invoked by application code:\n- `golang.org/x/sys` β€” low-level OS calls (file descriptor, signal handling)\n- `spf13/pflag` β€” flag parsing backend for cobra\n- `rogpeppe/go-internal` β€” internal tooling utilities\n\n### Summary\n\nThis file communicates **zero** information to the outside world at runtime. It is a static contract between module versions and Go compiler/toolchain expectations. All error handling for the *application* that uses these packages would live in `.go` source files, not here." - }, - "internal/config/env.go": { - "sha256": "74c71087c2e9c9e40f22b9105b3be0a1050313915cdb2d8810443df2f5dc70c4", - "facts": "#### [API_SIGNATURES]\n# Public Surface Area: `internal/config` package\n\n## Types\n\n### Structs\n\n| Name | Fields |\n|------|--------|\n| **ExtractionStep** | `Name` string, `Prompt` string |\n| **Config** | `ModelID` string, `OllamaBaseURL` string, `OllamaNumCtx` int, `DocsDir` string, `ExtractionSteps` []ExtractionStep, `Ignore` []string, `IgnoreExtensions` []string |\n\n---\n\n## Constants (exported)\n\n| Name | Type |\n|------|------|\n| `CodeReducerModelIdEnvKey` | string (`\"CODE_REDUCER_MODEL_ID\"`) |\n| `OllamaBaseUrlEnvKey` | string (`\"OLLAMA_BASE_URL\"`) |\n| `OllamaNumCtxEnvKey` | string (`\"OLLAMA_NUM_CTX\"`) |\n| `OllamaDefaultBaseURL` | string (`\"http://localhost:11434\"`) |\n| `OllamaDefaultModelID` | string (`\"ornith:9b\"`) |\n| `OllamaDefaultNumCtx` | int (`8192`) |\n| `ConfigFileName` | string (`.code-reducer.yaml`) |\n\n---\n\n## Functions\n\n### Public API\n\n| Function | Input | Output |\n|----------|-------|--------|\n| **getConfigPath** | `cwd` string | `string` |\n| **ConfigExists** | `cwd` string | `bool` |\n| **LoadConfig** | `cwd` string | `(*Config, error)` |\n| **SaveConfig** | `cwd` string, `cfg` *Config | `error` |\n| **MergeAndDeduplicate** | `a`, `b` []T (generic) | `[]T` |\n| **ResolveConfig** | `repoRoot`, `modelIdFlag`, `numCtxFlag` string | `(*Config, error)` |\n\n---\n\n## Package-level Variables (exported defaults)\n\n| Name | Type |\n|------|------|\n| `DefaultIgnores` | `[]string` |\n| `DefaultIgnoredExtensions` | `[]string` |\n| `DefaultExtractionSteps` | `[]ExtractionStep` |\n\n#### [BUSINESS_LOGIC]\nThis file defines the domain concept of **multi-layer configuration resolution** for a code analysis pipeline tool. It models how settings flow through four distinct layers: hard-coded defaults, YAML config files, environment variables, and CLI flagsβ€”each layer overwriting the previous one when values are present.\n\nThe algorithm resolves three key properties independently:\n1. **Model ID**: Defaults β†’ YAML `model_id` β†’ env `CODE_REDUCER_MODEL_ID` β†’ CLI flag (last wins)\n2. **Base URL**: Defaults β†’ YAML `ollama_base_url` β†’ env `OLLAMA_BASE_URL` (CLI flag intentionally omitted, suggesting it's not supported for this field)\n3. **Context Size**: Defaults β†’ YAML `ollama_num_ctx` β†’ env `OLLAMA_NUM_CTX` β†’ CLI flag (last wins), with validation that values must be positive integers\n\nThe file also defines a domain concept of **configurable analysis phases**. It structures code review into named extraction stepsβ€”API signatures, business logic, state/concurrency, and error handlingβ€”each with its own prompt template. These can be customized per project via YAML or replaced entirely by CLI flag.\n\nAdditionally, it handles **list deduplication** when merging user-provided ignore paths/extensions against system defaults, ensuring the same directory isn't listed twice regardless of source order.\n\n#### [STATE_AND_CONCURRENCY]\n**Mutable State:**\n\nThree package-level global variables exist, each holding a slice initialized at declaration time:\n\n| Variable | Type | Modified After Init? |\n|---|---|---|\n| `DefaultIgnores` | `[]string` | No β€” never modified after initialization in this file |\n| `DefaultIgnoredExtensions` | `[]string` | No β€” never modified after initialization in this file |\n| `DefaultExtractionSteps` | `[]ExtractionStep` | No β€” never modified after initialization in this file |\n\nAll other state (the `Config` struct fields, local variables like `resolved`, `resolvedIgnore`, etc.) is either:\n- Local to a function call (`ResolveConfig`) and returned as a value, or\n- Passed by pointer only for read/write within that single call scope.\n\n**Concurrency Mechanisms:**\n\nNone. There are no `sync.Mutex`, channels, atomic operations, locks, or any other concurrency primitives in this file or anywhere referenced in its API surface.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects and Error Handling Analysis\n\n### I/O Operations\n\n| Operation | Function | Direction | Details |\n|-----------|----------|-----------|---------|\n| `os.Stat` | `ConfigExists` | Read (disk) | Checks file existence. Swallows all errors (permission denied, I/O failure, etc.) β€” returns `false`, not a distinct error. |\n| `os.ReadFile` | `LoadConfig` | Read (disk) | Reads config file. If it fails, the raw error is returned to caller unchanged. |\n| `os.WriteFile` | `SaveConfig` | Write (disk) | Writes config atomically with `0600` permissions. Wraps any write error with `\"failed to write config file:\"`. |\n| `os.Getenv` | `ResolveConfig` | Read (env) | Reads three env vars (`CODE_REDUCER_MODEL_ID`, `OLLAMA_BASE_URL`, `OLLAMA_NUM_CTX`). Errors are swallowed β€” only non-empty values are used. |\n\n### Error Handling Patterns\n\n**1. Pass-through errors**\n- `LoadConfig`: The initial `os.ReadFile` error is returned as-is (no wrapping). Caller must handle file-not-found, permission denied, I/O failures uniformly.\n\n**2. Wrapped errors (all with `\"failed to %s:\"` prefix)**\n- YAML parse failure β†’ wrapped: `\"failed to parse yaml config: %w\"`\n- YAML marshal failure β†’ wrapped: `\"failed to marshal yaml: %w\"`\n- File write failure β†’ wrapped: `\"failed to write config file: %w\"`\n\n**3. Sentinel-based recovery (swallow + recover)**\n- `ResolveConfig`: After calling `LoadConfig`, checks `os.IsNotExist(err)`. If the error is \"file not exists\", it swallows it and initializes a fresh empty `Config{}`. All other errors propagate wrapped as `\"failed to load configuration file:\"`.\n\n**4. Conditional validation (error checked, value used only if valid)**\n- `OllamaNumCtx` env/flag resolution: `strconv.Atoi` result is accepted only when `err == nil \u0026\u0026 n \u003e 0`. Errors are swallowed; invalid values fall through and the previous resolved value (default/YAML/env) is retained.\n\n**5. No error return on success**\n- `SaveConfig` returns `nil` on success β€” no confirmation signal beyond a successful write.\n- `MergeAndDeduplicate[T comparable]` returns a new slice but never errors; duplicates are removed via map lookup, which silently ignores collisions.\n\n### Notable Observations\n\n- **Permission enforcement**: `SaveConfig` writes with `0600`. If the file already existed with different permissions, this call will fail (error wrapped). This is an implicit side effect β€” permission normalization on every write.\n- **Empty config fallback**: When config file is missing, `ResolveConfig` produces a default-configured struct rather than failing hard. The only way to get a non-nil error from `ResolveConfig` is if the YAML itself is malformed (parse wrap) or another unexpected I/O occurs during read.\n- **No atomic write guarantee**: `SaveConfig` does not write to a temp file then rename β€” it writes directly. A partial write failure could leave a truncated config on disk." - }, - "internal/engine/cache.go": { - "sha256": "ca70e4943ced77cd94ceee1a137a0d57c8127a82f1ea43bcfff5210c4b819bc0", - "facts": "#### [API_SIGNATURES]\n# Module: internal/engine (`cache.go`)\n\n## Exported Types\n\n### `FileCacheEntry` (struct)\n\n**Fields:**\n\n| Field | Type | JSON Tag |\n|-------|------|----------|\n| SHA256 | `string` | `\"sha256\"` |\n| Facts | `string` | `\"facts\"` |\n\n---\n\n### `MetadataCache` (struct)\n\n**Fields:**\n\n| Field | Type | JSON Tag |\n|-------|------|----------|\n| Files | `map[string]FileCacheEntry` | `\"files\"` |\n| Modules | `map[string]string` | `\"modules\"` |\n\n---\n\n## Exported Functions\n\n### `IsInitialized(repoRoot, docsDir string) bool`\n\n- **Parameters:**\n - `repoRoot` β€” `string`\n - `docsDir` β€” `string`\n- **Returns:** `bool`\n\n#### [BUSINESS_LOGIC]\n## Business Logic \u0026 Domain Concepts\n\n### Core Concept: Filesystem-Based State Cache\n\nThis module persists an in-memory cache to disk so that expensive computations can be avoided across runs. The cache is stored as `docsDir/.metadata.json` and holds two pieces of state:\n\n- **File-level metadata** (`Files map`): maps virtual paths (filenames) to cached SHA256 hashes plus associated \"facts\" strings.\n- **Module-level metadata** (`Modules map`): maps module identifiers to string values.\n\n### Algorithmic Flow\n\n1. **Load**: `loadMetadataCache()` attempts to read the JSON file. If absent, it returns a freshly initialized empty cache (not an error). If present but unparseable, it errors. Null maps are normalized to empty maps on load.\n2. **Check initialization**: `IsInitialized()` simply probes for the existence of `.metadata.json`. A successful read means the cache has been populated at least once.\n3. **Save**: `saveMetadataCache()` serializes the struct back to JSON with indentation and writes it atomically via `tools.WriteFileSafely`.\n4. **Hash computation**: `computeSHA256()` reads a file by its virtual path, runs SHA-256 on its content, and returns the hex-encoded digest.\n\n### Domain Interpretation\n\nThe cache structure suggests this engine tracks which files have been processed (via hash) and what facts were derived from them, alongside module-level state. The design treats \"not yet cached\" as a valid initial state rather than an error β€” meaning the first run always starts fresh.\n\n#### [STATE_AND_CONCURRENCY]\n## Mutable State\n\n- **`MetadataCache.Files`** (`map[string]FileCacheEntry`) β€” initialized with `make()` when nil; populated during `loadMetadataCache`/`saveMetadataCache`; serialized to `docsDir/.metadata.json`.\n- **`MetadataCache.Modules`** (`map[string]string`) β€” same pattern as above.\n\nBoth maps are mutated across the load/save lifecycle and persist on disk via JSON serialization, but no in-memory locking is applied within this file.\n\n## Concurrency Mechanisms\n\nNo concurrency primitives (sync.Mutex, channels, atomic operations) appear in this file. No mutable state is protected by any lock visible here.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects \u0026 Error Handling Analysis β€” `cache.go`\n\n---\n\n### 1. Disk I/O (the only outside-world communication)\n\nAll four functions communicate with the filesystem through a single relative path:\n\n| Function | Path constructed | Direction |\n|----------|-----------------|-----------|\n| `loadMetadataCache` | `docsDir/.metadata.json` | Read |\n| `IsInitialized` | `docsDir/.metadata.json` | Read (existence probe) |\n| `saveMetadataCache` | `docsDir/.metadata.json` | Write |\n| `computeSHA256` | `virtualPath` (caller-supplied) | Read |\n\nThe actual read/write operations are delegated to `tools.ReadFileSafely` and `tools.WriteFileSafely` from the external package. This means:\n\n- **No direct OS calls** β€” all I/O is funneled through a safety wrapper in another package.\n- The SHA256 computation itself is purely in-memory (`crypto/sha256`); no network or DB access occurs anywhere in this file.\n\n---\n\n### 2. Error handling patterns\n\n| Function | On missing cache file | On generic read/write error | On JSON parse error |\n|----------|----------------------|----------------------------|--------------------|\n| `loadMetadataCache` | Returns empty initialized `*MetadataCache`, `nil` error | Wraps with `\"failed to read metadata cache: %w\"` | Wraps with `\"failed to unmarshal metadata cache: %w\"` |\n| `IsInitialized` | N/A (returns `false`) | N/A β€” any non-nil error β†’ `false` | β€” |\n| `saveMetadataCache` | β€” | Returns the raw error from `tools.WriteFileSafely` | Returns the raw error from `json.MarshalIndent` |\n| `computeSHA256` | β€” | Returns the raw error from `tools.ReadFileSafely` | N/A (no parsing) |\n\n**Key observations:**\n\n- **No panic usage.** All errors are returned to callers.\n- **Error wrapping is selective.** Only `loadMetadataCache` wraps read/unmarshal failures with context strings; `saveMetadataCache` and `computeSHA256` pass raw errors through unchanged. This means the original error type from `tools.ReadFileSafely`/`WriteFileSafely` is preserved for those two functions, but lost (replaced by a new error) in `loadMetadataCache`.\n- **No sentinel types.** There are no custom error structs or wrapped error sentinels defined here. All errors are either raw I/O errors from the tools package or context-wrapped strings.\n- **Nil-safe JSON init.** Both `Files` and `Modules` maps inside `MetadataCache` are initialized to empty maps if their JSON values come back as `nil`, preventing potential nil-map dereferences downstream when accessing `.Files` or `.Modules`." - }, - "internal/engine/chunking.go": { - "sha256": "c40e8fda3be85e0f0957418fe9fdc18b81b3f85f76e0b2f097d96705cd171d79", - "facts": "#### [API_SIGNATURES]\n# Public Surface Area: `internal/engine/chunking.go` (package `engine`)\n\n**Exported items:** None.\n\nAll four functions in this file begin with a lowercase letter (`reduceInChunks`, `reduceFileFacts`, `reduceItems`, `chunkTextWithOverlap`), making them unexported to consumers of the package. No exported types, interfaces, or methods exist.\n\n#### [BUSINESS_LOGIC]\n## Domain Concepts \u0026 Business Rules\n\n### 1. Context-Aware Chunking Strategy\n\nThe system processes extracted code/fact items in batches constrained by LLM context limits (`NumCtx * 3`). This prevents exceeding token budgets while preserving information integrity. The chunking logic adapts dynamically: if individual items can't fit together within the limit, each item gets truncated to a per-item allowance; otherwise, items are grouped into logical batches before synthesis.\n\n### 2. Two Distinct Processing Modes\n\n**Module Synthesis (`reduceInChunks`)** \nFor architectural-level nodes, facts are synthesized into architecture descriptions using a \"module_synthesis\" system prompt. This mode assumes each batch represents evidence contributing to understanding the module's design intent.\n\n**File Fact Consolidation (`reduceFileFacts`)** \nFor file-level analysis, extracted facts about a specific step within a file are consolidated with deduplication and merging. A specialized base system prompt instructs the LLM to act as a code documentation assistant focused on producing cohesive summaries from fragmented evidence.\n\n### 3. Recursive Convergence Algorithm (`reduceItems`)\n\nThe core business rule: reduce items in batches β†’ collect intermediate results β†’ recursively apply reduction until everything converges into a single output string. This multi-pass approach ensures that information is progressively synthesized rather than lost across multiple LLM calls. The recursion terminates when all data fits within one batch, guaranteeing eventual convergence even for large inputs.\n\n### 4. Graceful Truncation Policy\n\nWhen content exceeds the context window:\n- Single oversized items get truncated with a `...[truncated]` marker appended\n- When multiple items exceed limits collectively, each gets capped at `maxChars / len(items)` characters before being processed as one batch\n\nThis preserves signal that information was intentionally shortened rather than silently dropped.\n\n### 5. Overlapping Chunk Support (`chunkTextWithOverlap`)\n\nA utility for splitting raw text into overlapping chunks to maintain context continuity between adjacent segmentsβ€”useful when feeding source material directly into LLMs without pre-processing through the chunking logic.\n\n#### [STATE_AND_CONCURRENCY]\n**Mutable State:** None in this file. All functions operate on parameters and return values; no global variables are declared or used. The `items[0] = ...` assignment inside `reduceItems` mutates only the caller's local slice argument, not any shared state.\n\n**Concurrency Mechanisms:** None. No channels, mutexes, atomic operations, or other concurrency primitives are present in this file.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects (External Communication)\n\n| Function | External I/O | Notes |\n|---|---|---|\n| `reduceInChunks` | **Network** β€” calls `c.CallLLM(ctx, ...)` which communicates with an LLM service | Prompt is built via `fmt.Sprintf`; system prompt fetched from `c.GetDefaultSystemPrompt(\"module_synthesis\")`. Response goes through `StripOuterMarkdownFence`. |\n| `reduceFileFacts` | **Network** β€” calls `c.CallLLM(ctx, ...)` which communicates with an LLM service | System prompt is built locally by concatenating `c.GetBaseSystemPrompt()` + a fixed role string. Response also passes through `StripOuterMarkdownFence`. |\n| `reduceItems` | None visible in source text | Recursion only; no I/O calls. |\n| `chunkTextWithOverlap` | None visible in source text | Pure rune/string manipulation. |\n\n**Unresolved:** The actual transport, protocol, and target of `c.CallLLM()` is not observable from this file alone (it lives on the `*LLMClient` type).\n\n## Error Handling\n\n### Returned Errors\n- Both LLM calls (`reduceInChunks`, `reduceFileFacts`) wrap errors with a descriptive prefix using `%w` for chain preservation:\n - `\"LLM error during synthesis: %w\"` \n - `\"LLM error during file fact consolidation: %w\"`\n- Recursive calls in `reduceItems` propagate the inner error straight up: `return \"\", err`.\n\n### Silent Swallowing (no panic, no returned error)\n- **Empty input** (`len(items) == 0`) β†’ returns `\"\", nil`\n- **Single-item truncation** when one item exceeds `maxChars` runes β†’ silently truncated with suffix `\\n...[truncated]` and a newline appended.\n- **Infinite-loop guard**: if batch size equals item count for all items (can't merge), falls back to per-item truncation using `allowedPerItem = maxChars / len(items)`. Still silent β€” no error returned.\n- **`chunkTextWithOverlap`** with `maxRunes \u003c= 0` β†’ returns `[]string{text}` (full text as single chunk).\n- **`chunkTextWithOverlap`** with `overlapRunes \u003e= maxRunes` β†’ clamps to half without warning.\n- **`chunkTextWithOverlap`** when the final partial chunk reaches end of runes β†’ breaks loop; no error, no truncation marker (unlike `reduceItems`).\n\n### Recursion Termination\n`reduceItems` terminates because:\n1. Single-batch case short-circuits at top (`len(batches) == 1`).\n2. Recursive step processes each batch through itself again with the same `maxChars`. Errors from deeper recursion propagate up; otherwise, intermediate results are re-processed until a single result emerges or an error is returned." - }, - "internal/engine/client.go": { - "sha256": "320c60925f09809eea8a8f27d61551a9667f48b9cca3d70434d03b1322d0f75b", - "facts": "#### [API_SIGNATURES]\n# Public Surface Area of `client.go` (Module: `internal/engine`)\n\n## Exported Types\n\n### `Message`\n\n```go\ntype Message struct {\n Role string `json:\"role\"`\n Content string `json:\"content\"`\n}\n```\n\n---\n\n### `LLMClient`\n\n```go\ntype LLMClient struct {\n ModelID string\n BaseURL string\n NumCtx int\n HTTPClient *http.Client\n}\n```\n\n---\n\n## Exported Methods (on `*LLMClient`)\n\n### `NewLLMClient`\n\n| Parameter | Type | Description |\n|-----------|----------|---------------------|\n| modelID | string | LLM identifier |\n| baseURL | string | API base URL |\n| numCtx | int | Context window size |\n\n**Returns:** `*LLMClient` β€” initialized client with a 10-minute HTTP timeout.\n\n---\n\n### `CallLLM`\n\n```go\nfunc (c *LLMClient) CallLLM(ctx context.Context, systemPrompt string, messages []Message, jsonFormat bool) (string, error)\n```\n\n| Parameter | Type | Description |\n|-----------|-------------|------------------------------|\n| ctx | `context.Context` | Cancellation/timeout |\n| systemPrompt | string | System prompt to prepend |\n| messages | `[]Message` | User/conversation messages |\n| jsonFormat | bool | Whether response should be JSON |\n\n**Returns:** `(string, error)` β€” the LLM's response content (or empty string on failure).\n\n---\n\n### `GetBaseSystemPrompt`\n\n```go\nfunc (c *LLMClient) GetBaseSystemPrompt() string\n```\n\n**Returns:** `string` β€” the base Code-Reducer system prompt.\n\n---\n\n### `GetDefaultSystemPrompt`\n\n```go\nfunc (c *LLMClient) GetDefaultSystemPrompt(command string) string\n```\n\n| Parameter | Type | Description |\n|-----------|----------|----------------------|\n| command | string | One of: `\"module_synthesis\"`, `\"architecture\"`, any other value |\n\n**Returns:** `string` β€” the composed system prompt for the given command.\n\n#### [BUSINESS_LOGIC]\n## Domain Concepts \u0026 Business Rules\n\n### Core Responsibility\nThis file implements an **LLM client abstraction** β€” specifically targeting Ollama-compatible inference endpoints. It encapsulates the full lifecycle of sending prompts, receiving responses, and returning structured text output.\n\n### Data Flow\n1. A user or orchestrator provides a system prompt plus one or more user/assistant messages.\n2. The client prepends the system prompt as the first message in the conversation array (role: \"system\").\n3. All messages are serialized into an Ollama `/api/chat` JSON payload with model ID, context window size, and stream flag.\n4. A non-streaming HTTP POST request is dispatched; the response body is read once and parsed as `{\"message\":{\"content\":\"...\"}}`.\n5. On success, only the assistant content string is returned to the caller. On failure (non-200 status), a truncated error message containing the status code is returned.\n\n### Prompt Composition Rules\nA layered prompt system exists:\n- **Base layer**: A fixed role definition and defensive guidelines that every LLM call inherits regardless of task type.\n- **Task layer**: Command-specific instructions are appended based on the command string. Two named variants exist (`module_synthesis`, `architecture`); any other command receives only the base layer.\n\n### Key Business Constraints\n- The client does not retry failed requests β€” errors propagate directly to the caller.\n- Streaming is disabled by design (the `CallLLM` method hardcodes `stream = false`).\n- Context window size is configured at construction time and applied uniformly across all calls for that client instance.\n\n#### [STATE_AND_CONCURRENCY]\n## State Mutation \u0026 Concurrency Analysis\n\n**File:** `client.go` (Module: internal/engine)\n\n### Mutable State: None detected.\n\n- The `LLMClient` struct fields (`ModelID`, `BaseURL`, `NumCtx`, `HTTPClient`) are set once in the constructor and never reassigned within any method.\n- All methods operate on local variables or create new slices/structures without modifying receiver state.\n- No global package-level mutable variables are declared or modified.\n\n### Concurrency Mechanisms: None detected.\n\n- No `sync.Mutex`, channels, atomic operations, or other synchronization primitives are used in this file.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## External Communication (I/O)\n\n**Network I/O only.** The code communicates exclusively via HTTP to an Ollama-compatible API endpoint:\n\n| Function | Endpoint | Method | Transport | Timeout |\n|---|---|---|---|---|\n| `CallLLM` β†’ `prepareOllamaRequest` β†’ `c.HTTPClient.Do(req)` | `{BaseURL}/api/chat` | POST | Standard library `http.Client` | 10 minutes (`NewLLMClient`) |\n\n- No disk, DB, or file-system access.\n- The default HTTP transport is created inline with no custom configuration (no proxy, no redirect policy, no TLS config).\n- Request headers are set to `application/json`.\n- Response body is fully read on 200 OK; on non-OK status codes the body is truncated at 1 KB via `io.LimitReader(resp.Body, 1024)`.\n\n## Error Handling / Return Values\n\n**No panics.** All errors are returned as typed values to the caller.\n\n### Per-function error behavior:\n\n| Function | Failure Path | Error Type | Wrapping Strategy |\n|---|---|---|---|\n| `NewLLMClient` | None (constructor) | β€” | β€” |\n| `prepareOllamaRequest` | JSON marshal fails | `fmt.Errorf(\"failed to marshal request: %w\", err)` | Wrapped with context (`%w`) |\n| `prepareOllamaRequest` | `http.NewRequestWithContext` fails | Returns raw error from `NewRequestWithContext` | **Not wrapped** β€” passed through as-is |\n| `CallLLM` β†’ HTTP Do | Network-level failure (DNS, conn reset, timeout, etc.) | Returns raw error from `c.HTTPClient.Do(req)` | **Not wrapped** β€” passed through as-is |\n| `CallLLM` β†’ 200 OK path | Body read fails (`io.ReadAll`) | Returns raw error from `io.ReadAll(resp.Body)` | **Not wrapped** |\n| `CallLLM` β†’ 200 OK path | JSON unmarshal fails | `fmt.Errorf(\"failed to parse response: %w\", err)` | Wrapped with context (`%w`) |\n| `CallLLM` β†’ non-OK status | Response body truncated at 1 KB for size safety | `fmt.Errorf(\"ollama api error: status %d, response: %s\", resp.StatusCode, string(body))` | New error constructed (not wrapped). Status code + partial body included. |\n\n### Notable patterns:\n\n- **Non-OK path discards the first return value** of `io.ReadAll(io.LimitReader(...), 1024)` β€” only the second value (the truncated bytes) is used, and only if reading succeeded; otherwise `string(body)` would be an empty string. This means on a read failure in the error-status path, the returned error message will say `\"ollama api error: status \u003ccode\u003e, response: \"` with no body detail.\n- **No error wrapping for transport-level failures.** Network errors from `http.Client.Do` and request-construction errors are returned verbatim to callers. Callers must inspect the underlying error (e.g., check for timeout, DNS failure) themselves if needed." - }, - "internal/engine/json_parser.go": { - "sha256": "972f1e04f6c1529ab3be5ec00ea152a3bd9874b323118ba148b782b8ba41d4e2", - "facts": "#### [API_SIGNATURES]\n# Public Surface Area of `internal/engine/json_parser.go`\n\n## Functions\n\n| Function | Input Types | Output Type(s) |\n|---|---|---|\n| **StripOuterMarkdownFence** | `string` | `string` |\n| **CleanJSONResponse** | `string` | `string` |\n| **UnmarshalJSONResponse** | `string`, `interface{}` | `error` |\n\n## Notes\n\n- No structs, interfaces, or methods are defined in this file.\n- Internal function `extractBalancedJSON` is not exported (lowercase first letter) and excluded from the public surface area.\n\n#### [BUSINESS_LOGIC]\n**Domain Concept:** This file implements an **LLM response parser** designed to extract structured data from raw model outputs. It handles the reality that LLM responses often wrap JSON inside markdown code fences, may contain multiple JSON objects, or include malformed content.\n\n**Business Rules:**\n\n1. Markdown code fences (` ``` `) surrounding JSON must be stripped before parsing.\n2. Balanced bracket matching is required β€” strings containing `{`, `[`, `}`, `]` literals (including escaped characters like `\\n`) must not break extraction logic.\n3. When multiple balanced JSON candidates exist in a single response, each must be attempted individually until one succeeds, or the entire trimmed string is tried as a final fallback.\n\n**High-Level Algorithmic Flow:**\n\n1. Strip surrounding markdown fences from the raw input.\n2. Scan for opening brackets `{` or `[`, then extract the longest balanced bracket sequence starting at that position (respecting string literals).\n3. Collect all valid balanced-candidate strings found in step 2.\n4. Attempt `json.Unmarshal` on each candidate; return the first success.\n5. If no candidate succeeds, attempt unmarshaling the entire trimmed input as a final fallback.\n6. Return an error with context if all attempts fail.\n\n**Key Design Choice:** The balanced-extraction logic operates at the rune level and tracks escape state inside string literals so that JSON embedded within markdown formatting or prose is correctly isolated rather than truncated at the first unmatched bracket.\n\n#### [STATE_AND_CONCURRENCY]\n**Mutable State:** None. The code uses only stack-local variables (`stack`, `inString`, `escape` in `extractBalancedJSON`) and string parameters. All functions are pure transformations on their inputs with no shared mutable state.\n\n**Concurrency Mechanisms:** None. The file contains no `sync.Mutex`, channels, or any other concurrency primitives.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects (I/O)\n\n**None.** This file performs zero external communication:\n- No network calls, no disk reads/writes, no database access.\n- Only imports from the standard library (`encoding/json`, `fmt`, `regexp`, `strings`).\n- All functions operate entirely on in-memory strings and byte slices.\n\n## Error Handling\n\n### `StripOuterMarkdownFence` β€” silent swallow\n- Returns only a string; never returns an error.\n- If no markdown fence matches, it falls back to returning the trimmed input unchanged. No error is surfaced.\n\n### `extractBalancedJSON` β€” explicit errors\n- Returns `(string, error)`.\n- Two distinct error paths:\n - `\"unbalanced closing delimiter\"` β€” raised when a closing brace/bracket appears without a matching opening one on the stack.\n - `\"unbalanced delimiters\"` β€” returned at the end of the loop if the full input was scanned without ever completing a balanced pair.\n\n### `CleanJSONResponse` β€” error swallowed internally\n- Returns only a string; callers get no signal that an unbalanced JSON candidate was encountered.\n- It iterates through the stripped content, calls `extractBalancedJSON`, and on any successful candidate returns it immediately. If no candidate is found, it returns the original trimmed input unchanged.\n\n### `UnmarshalJSONResponse` β€” structured fallback chain\n- Returns only an error; uses a multi-layered attempt strategy:\n 1. **Candidate sweep** β€” collects every balanced JSON substring (`{...}` or `[...]`) and tries to unmarshal each into `target`. Keeps the last error in `lastErr` for later use.\n 2. **Whole-string fallback** β€” if any candidate succeeded, returns `nil`; otherwise attempts `json.Unmarshal` on the entire trimmed string.\n 3. **Terminal errors** β€” if both layers fail:\n - Returns `\"failed to unmarshal JSON candidates: %w\"` wrapping the last candidate error (if one was captured).\n - If no candidates were ever collected, returns `\"no valid JSON found in response\"`.\n\n### Summary of caller-visible behavior\n| Function | Error surface | What gets swallowed |\n|---|---|---|\n| `StripOuterMarkdownFence` | none | mismatched / missing fences |\n| `extractBalancedJSON` | explicit (`error`) | β€” |\n| `CleanJSONResponse` | none (string only) | unbalanced JSON inside the response |\n| `UnmarshalJSONResponse` | explicit (`error`) | partial parse failures are collected and re-emitted via `%w`; a \"no valid JSON\" path covers the case where nothing was attempted. |" - }, - "internal/engine/orchestrator.go": { - "sha256": "f7c0c09caedfc0a520b339a6c994900335500979e95dc2d212ad38023c811dd6", - "facts": "#### [API_SIGNATURES]\n# Public Surface Area: `orchestrator.go` (Module: `internal/engine`)\n\n## Structs\n\n### `Event`\n\n| Field | Type |\n|---|---|\n| `Type` | `string` |\n| `Message` | `string` |\n\n\u003e **Note:** All methods on `*LLMClient` are unexported (lowercase receiver). They belong to the external type `LLMClient`, not this file's public API.\n\n#### [BUSINESS_LOGIC]\n## Business Rules \u0026 Domain Concepts\n\n### Core Purpose\nThis file orchestrates an **LLM-driven documentation pipeline** that generates and maintains technical documentation for a codebase using a Map-Reduce pattern. The system produces three outputs: `architecture.md` (global overview), `quickstart.md` (onboarding guide), and per-module API summaries under `modules/`.\n\n### Domain Concepts\n\n| Concept | Description |\n|---|---|\n| **Code Discovery** | Recursive traversal of the repo tree, filtering by configurable ignore lists (.gitignore patterns + user-specified exclusions) and extension blacklists. |\n| **File Fingerprinting** | SHA-256 hashing of code files to detect additions, modifications, deletions between pipeline runs. |\n| **Metadata Cache** | Persists `Files` (path β†’ hash), `Modules` (directory paths with summaries), and the last root summary so subsequent runs can skip unchanged work. |\n| **Hierarchical Tree-Merging** | Builds a directory tree from discovered files, then recursively synthesizes each node's content via LLMβ€”bottom-up reduction of code into documentation. |\n| **Affected Module Propagation** | Marks directories whose contents changed, then walks the module summary cache and prunes summaries for untouched modules. Only affected modules are re-synthesized; global docs regenerate only when root-level changes occur or global files don't exist. |\n\n### Pipeline Flow β€” `RunInit` (Full Run)\n\n1. **Setup** β€” Load `.gitignore`, merge with user ignore list, load metadata cache from disk.\n2. **Map Phase** β€” Discover code files β†’ compute SHA-256 hashes for each β†’ build directory tree.\n3. **Mark All Affected** β€” Recursively flag every node as affected (init starts fresh).\n4. **Reduce Phase** β€” Call `synthesizeNode` bottom-up on the tree to produce a root summary string and per-module summaries via LLM.\n5. **Generate Standard Docs** β€” Produce `architecture.md` and `quickstart.md` from the root summary.\n6. **Update AGENTS.md** β€” Read existing file; if missing, create with guidelines header. If present but doesn't contain \"AI Agent Guidelines\", append a separator + new content block. Only writes when content actually differs.\n7. **Teardown** β€” Persist metadata cache to disk.\n\n### Pipeline Flow β€” `RunUpdate` (Incremental Run)\n\n1. **Setup** β€” Load pipeline state from metadata cache.\n2. **Map Phase** β€” Discover code files β†’ hash each β†’ build tree.\n3. **Change Detection** β€” Compare current file hashes against cached hashes:\n - New file in codebase but absent from cache β†’ `Added`.\n - Existing file with different hash β†’ `Modified`.\n - Cached file no longer in codebase β†’ `Deleted` (also removed from cache).\n4. **Prune Stale Modules** β€” Remove any module summaries whose directory is no longer active, and delete their markdown files on disk.\n5. **Propagate Affected** β€” Run tree-aware propagation to mark directories affected by changes.\n6. **Early Exit** β€” If zero directories are affected β†’ pipeline ends with \"up to date\" status.\n7. **Conditional Reduce** β€” If root-level change detected or global docs don't exist, run full synthesis; otherwise skip and reuse existing `architecture.md`/`quickstart.md`.\n8. **Teardown** β€” Save updated cache.\n\n### Key Business Rules\n\n- **Init is all-or-nothing**: Always regenerates everything from scratch regardless of cache state.\n- **Update is incremental**: Only re-generates documentation for directories whose contents actually changed. Unchanged modules keep their summaries intact.\n- **Global docs are conditional**: `architecture.md` and `quickstart.md` only regenerate when at least one top-level directory is affected or they don't exist on disk. Otherwise, existing content is preserved.\n- **AGENTS.md is append-only with deduplication**: Existing guidelines block prevents duplicate headers; new content appends after the original block.\n- **Cache is best-effort**: Failures to load/save cache are logged as warnings but do not halt the pipeline.\n\n#### [STATE_AND_CONCURRENCY]\n## Mutable State Analysis: `orchestrator.go` (Module: internal/engine)\n\n### Mutable State\n\n| Variable | Scope | Mutated By | Description |\n|---|---|---|---|\n| `cache.Files` | Local (`setupPipeline`/`RunInit`/`RunUpdate`) | `synthesizeNode` β†’ `updateCacheFromHashes`, `updateModulesTree`, `deleteStaleFiles` | Map[string]*FileMetadata β€” populated in `RunInit`; modified during synthesis in `RunUpdate`. |\n| `cache.Modules` | Local (`setupPipeline`/`RunInit`/`RunUpdate`) | `synthesizeNode` β†’ `buildModuleMap`, `updateModulesTree`, `deleteStaleFiles` | Map[string]*ModuleMetadata β€” populated via `modulesDir` in `RunInit`; modified during synthesis in `RunUpdate`. |\n| `precalculatedHashes` | Local (`RunInit`) | Loop iterates over `codeFiles` and inserts hash values. | Map[string]string β€” local scratch map, not shared. |\n| `affectedDirs` | Local (`RunInit`/`RunUpdate`) | Recursive closures: `markAllAffected`, `collectDirs` (also used in `determineAffected`). | Map[string]bool β€” populated by recursive traversal of tree nodes; passed to `synthesizeNode`. |\n| Files on disk (`architecture.md`, `quickstart.md`, `AGENTS.md`, module files) | OS filesystem | `RunInit`, `RunUpdate`, `GenerateStandardDocs` | Written via `tools.WriteFileSafely`; `teardownPipeline` writes the cache back to disk. |\n\n### Concurrency Mechanisms\n\n**No concurrency mechanisms are used.** No `sync.Mutex`, channels, atomic operations, or other synchronization primitives appear in this file. All mutable state is local to function invocations and there is no shared mutable state across goroutines in this code.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects \u0026 Error Handling Analysis: `orchestrator.go`\n\n---\n\n### 1. External Communication (I/O)\n\n#### LLM Calls (`c.CallLLM`)\n- **`RunInit`** β†’ calls `synthesizeNode(ctx, c, tree, ...)` which internally invokes `CallLLM`. \n- **`GenerateStandardDocs`** β†’ calls `CallLLM` twice: once for architecture.md, once for quickstart.md. \n- **`RunUpdate`** β†’ conditionally calls `synthesizeNode(...)` (via LLM) and optionally re-runs `GenerateStandardDocs` (two more LLM calls).\n\n#### Filesystem I/O\n| Operation | Function | Purpose |\n|---|---|---|\n| Read `.gitignore` | `tools.LoadGitignore(repoRoot)` | Loads ignore patterns at pipeline start. |\n| Scan repo for code files | `tools.DiscoverCodeFiles(...)` | Discovers source files; used by both pipelines. |\n| Compute SHA256 per file | `computeSHA256(repoRoot, f)` | Reads each file to hash it. Errors silently skipped in loops. |\n| Build directory tree | `buildTree(codeFiles)` | Constructs a `DirNode` tree from discovered paths. No I/O. |\n| Create modules dir | `os.MkdirAll(modulesDir, 0755)` | Creates `docsDir/modules/`. Wrapped error if fails. |\n| Read existing AGENTS.md | `tools.ReadFileSafely(repoRoot, \"AGENTS.md\")` | Reads current content (if any). |\n| Write new AGENTS.md | `tools.WriteFileSafely(...)` | Writes full or appended content. Wrapped error if write fails after read failure. |\n| Check file existence | `os.Stat(absArch)`, `os.Stat(absQs)` | Used in `RunUpdate` to decide whether to regenerate global docs. |\n| Remove stale module files | `os.Remove(absModuleFile)` | Cleans up orphaned module markdown files during update. Error swallowed (`_ =`). |\n\n#### Path Resolution (`security.SafeResolve`)\n- Used for safe path resolution before file operations (arch, quickstart, modules). No network I/Oβ€”pure filesystem path handling.\n\n#### Event Channel (`logEvent` callback)\n- `RunInit` and `RunUpdate` both wrap the user-supplied `onEvent func(Event)` into a local `logEvent(string, string)` helper that emits `{Type: \"status\", Message: ...}` events. \n- This is how progress/status messages leave the function (e.g., step markers). Errors are **not** sent through this channelβ€”only status strings flow here.\n\n---\n\n### 2. Error Handling Patterns\n\n#### Errors Returned to Caller\nThese propagate up unmodified or wrapped with a prefix:\n- `tools.DiscoverCodeFiles(...)` β†’ raw error returned if file discovery fails. \n- `synthesizeNode(ctx, c, tree, ...)` β†’ raw error returned on LLM failure. \n- `c.GenerateStandardDocs(...)` β†’ wrapped errors for architecture/quickstart generation and disk writes (e.g., `\"failed to write quickstart.md: %w\"`). \n- `os.MkdirAll(modulesDir, 0755)` β†’ wrapped with `\"failed to create modules directory: %w\"`. \n- AGENTS.md write chain β†’ if read fails but write succeeds, returns `\"failed to write AGENTS.md: %w\"`. If both fail, errors wrap each other.\n\n#### Errors Swallowed (logged but not returned)\nThese are caught and logged as warnings via `logEvent`, then execution continues:\n- **`tools.LoadGitignore(...)`** β†’ logs warning, ignores error. \n- **`loadMetadataCache(repoRoot, docsDir)`** β†’ logs warning, ignores error. \n- **`saveMetadataCache(repoRoot, docsDir, cache)`** in `teardownPipeline` β†’ logs warning, function always returns `nil` at end regardless of this failure. \n- **`computeSHA256(...)` errors** in both pipelines β†’ silently skipped (only stored if hash computed successfully).\n\n#### Errors Silently Discarded\n- **`os.Remove(absModuleFile)`** during module cleanup in `RunUpdate` β†’ error assigned to `_`, never returned or logged. \n- **`determineAffected(tree, ...)` and `propagateAffected(tree, ...)`** β†’ called without capturing return values; any errors inside are lost.\n\n#### No Panics\n- Zero `panic()` calls found. All failures use the standard `error` interface with `return err`.\n\n---\n\n### 3. Summary Table\n\n| Aspect | Detail |\n|---|---|\n| **I/O targets** | LLM service (via `CallLLM`), local filesystem under `repoRoot` (md files, cache, dir tree). No DB or network beyond LLM. |\n| **Error propagation** | Mixed: some paths wrap errors (`fmt.Errorf(\"...: %w\", err)`); others return raw errors; several paths swallow and log only. |\n| **Sentinel types** | None. All errors are generic `error` interface values. |\n| **Panic handling** | None present in this file. |\n| **Side-effect sink** | `logEvent(string, string)` callback for status messages; no error channel equivalent exists. |" - }, - "internal/engine/runner.go": { - "sha256": "50177f41ec89d76e503c57760f5f77f32cd49a4476e3c06d76314cce95577116", - "facts": "#### [API_SIGNATURES]\n## Public Surface Area: `internal/engine/runner.go`\n\n### Structs\n\n- **`Runner`** β€” exported struct containing a single field `cfg *config.Config`.\n\n---\n\n### Functions / Methods\n\n| Signature | Input Types | Output Type |\n|-----------|------------|-------------|\n| `NewRunner(cfg *config.Config)` | `*config.Config` | `*Runner` |\n| `(r *Runner).Run(ctx context.Context, repoRoot string, mode string, onEvent func(Event))` | `context.Context`, `string`, `string`, `func(Event)` | `error` |\n\n---\n\n### Notes\n\n- **No interfaces** are defined in this file.\n- The parameter type `func(Event)` references a type `Event` that is not defined within this file; its definition originates from an external package (not importable from the imports shown). I cannot confirm whether it is exported or where it lives, so I omit it as an independently reportable item.\n- Internal helper functions (`logEvent`, etc.) are unexported and omitted.\n\n#### [BUSINESS_LOGIC]\n**Business Rules / Domain Concepts:**\n\n1. **Repository isolation via lockfile**: Before any work begins, a `.gitignore` entry for the lockfile is ensured (best-effort; failures are logged as warnings and continue). After the run completes, the lock must be releasedβ€”this enforces single-concurrent execution per repository root.\n\n2. **Concurrent access control**: A repository-level lock is acquired before any LLM work starts. If the lock cannot be obtained, the operation fails outright. This prevents parallel runs from interfering with each other's state or prompts. The defer ensures cleanup regardless of how the run terminates.\n\n3. **Mode-gated pipeline selection**: The runner dispatches to either `init` or `update` based on a string parameter. Unknown modes are rejected immediately. Each mode invokes its own pipeline method on the LLM client, which is constructed from config (model ID, base URL, context size).\n\n4. **Event-driven observability**: An optional callback receives typed events throughout execution. The runner itself emits at least two event types: a warning for lockfile gitignore failures and informational status events. This lets downstream consumers observe progress without the engine needing to know their identity.\n\n**High-Level Flow:**\n\n```\nRun()\n β”œβ”€ ensure .gitignore entry (best-effort)\n β”œβ”€ acquire repo lock (fail-fast on conflict)\n β”œβ”€ build LLM client from config\n β”œβ”€ if mode == \"init\" β†’ RunInit pipeline\n β”œβ”€ if mode == \"update\" β†’ RunUpdate pipeline\n └─ else β†’ reject with unsupported-mode error\n```\n\nThe flow is sequential and linear: file-system prep β†’ concurrency guard β†’ model setup β†’ branch by mode. The lock is released automatically via defer once the run finishes, whether success or failure.\n\n#### [STATE_AND_CONCURRENCY]\n### Mutable State\n\n- **`cfg *config.Config`** (field on `Runner`) β€” Set once in `NewRunner`. Not mutated anywhere within this file; only read through methods delegated to other packages (`client.RunInit`, `client.RunUpdate`). External mutation is possible before construction, but no writes occur here.\n- **`lock`** (local variable) β€” Acquired via `security.AcquireLock(repoRoot)` and released via `defer lock.Unlock()`. Local scope only; not persisted on the struct.\n\n### Concurrency Mechanisms\n\n- No `sync.Mutex`, channels (`chan`), `atomic.*`, or other concurrency primitives are used in this file.\n- The repository lock (`lock`) is acquired and held for the duration of `Run`; however, its internal implementation (what type it is) is not visible here. Cannot confirm whether it provides mutual exclusion without inspecting `security.AcquireLock`.\n\n### Summary\n\n**No mutable state.** β€” All struct fields are set once at construction; no writes occur within this file. The only lock (`lock`) is a local variable with no concurrency primitive observable from this source alone.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects (Outside-World Communication)\n\n| # | Call | Effect on Outside World | Notes |\n|---|------|------------------------|--------|\n| 1 | `security.EnsureGitignoreHasLockfile(repoRoot)` | **Disk write** β€” modifies `.gitignore` in the repo root to add a lockfile entry. | If it fails, only logs an event (`logEvent(\"status\", ...)`). No error returned; execution continues. |\n| 2 | `security.AcquireLock(repoRoot)` | **Disk write** β€” creates or updates a repository lock file. Returns handle with `Unlock()` for deferred cleanup. | On failure, wrapped as `\"failed to acquire repository lock\"` and returned. |\n| 3 | `client.RunInit(ctx, repoRoot, r.cfg, onEvent)` | **Network + Disk** β€” calls LLM/Ollama endpoint (inferred from `OllamaBaseURL`/`OllamaNumCtx` config), writes documentation artifacts to disk. Fires events via `onEvent`. | Errors propagated as-is. |\n| 4 | `client.RunUpdate(ctx, repoRoot, r.cfg, onEvent)` | **Network + Disk** β€” same category as init, but for update mode. Fires events via `onEvent`. | Errors propagated as-is. |\n| 5 | `logEvent(Event{...})` | **In-memory callback only**. Does not write to disk/network itself. The event payload may be consumed elsewhere (e.g., persisted by another subsystem), but this code cannot determine that from its text alone. | N/A β€” purely internal. |\n\n## Error Handling Strategy\n\n| # | Pattern | Where |\n|---|---------|-------|\n| 1 | **Wrapped error** (`fmt.Errorf(\"...\", err)`) | Lock acquisition failure β†’ `\"failed to acquire repository lock: %w\"`. Caller sees the original cause via `errors.Is`/`errors.As`. |\n| 2 | **Propagated directly** (no wrap) | `RunInit`, `RunUpdate` failures β†’ returned as-is from `Run()`. |\n| 3 | **Wrapped error** (`fmt.Errorf(\"...\", mode)`) | Unsupported mode β†’ `\"unsupported mode: %s\"`. Caller sees the bad input value. |\n| 4 | **Swallowed silently** (logged event only, no return) | Gitignore enforcement failure. The warning is emitted via `logEvent`, but execution continues and `Run()` returns `nil` if nothing else fails after this point. |\n| 5 | **Deferred cleanup** (`defer lock.Unlock()`) | Acquired lock released automatically on any code path after it succeeds, including error paths in the init/update branches. |\n\n## Unhandled Paths (No Panic Recovery Visible)\n\n- Any panic raised inside `RunInit`, `RunUpdate`, or deeper in `security`/LLM client packages propagates out of `Run()` as a stack trace to the caller β€” no recovery here.\n- Context cancellation (`ctx.Done()`) is accepted but not explicitly checked; whichever underlying operation observes it will return its own error (likely via `context.DeadlineExceeded`)." - }, - "internal/engine/synthesize.go": { - "sha256": "7822c30124716956dc9fad7aab5756ede1ee371e490e8ee5883398a45b5f60d6", - "facts": "#### [API_SIGNATURES]\n## Public Surface Area Analysis\n\n```file:///internal/engine/synthesize.go\n```\n\n**Package:** `engine` (module: `internal/engine`)\n\n### Exported Types / Interfaces / Methods\n\n\u003e **None.** This file contains zero exported symbols.\n\nThe only function defined in this file is `synthesizeNode`, which begins with a lowercase letter and is therefore unexported (private to the package).\n\n#### [BUSINESS_LOGIC]\n## Business Logic \u0026 Domain Concepts\n\n### Core Purpose\nThis function performs recursive synthesis of directory structures into hierarchical summariesβ€”transforming raw source code (files) and subdirectory metadata into consolidated documentation at each level of a module tree.\n\n---\n\n### Algorithmic Flow\n\n**1. Context Check β†’ Cache Reuse Gate** \nIf the current node is not in `affectedDirs` AND a cached module summary exists, return immediately without any processing. This short-circuits unaffected directories.\n\n**2. Recursive Child Synthesis (Bottom-Up)** \nSort children alphabetically, then recursively call `synthesizeNode` on each child directory. Results are collected into a map keyed by child name.\n\n**3. File Processing Loop** \nFor each file in the current node:\n- **Hash-first cache lookup**: If a precalculated hash exists and matches the cached SHA256, reuse cached facts without reading disk.\n- **Read + Hash fallback**: Otherwise read the file, compute its SHA256 hash, update the cache entry if it already existed with that hash.\n- **Extraction pipeline** (when no facts found):\n - Chunk the raw content using a dynamic limit derived from `c.NumCtx Γ— 4 characters Γ— 0.75` (with a minimum of 512 tokens and a max overlap cap).\n - For each chunk, iterate over every step in `cfg.ExtractionSteps`, sending each chunk to the LLM with a system prompt built from base + step-specific prompt.\n - Consolidate all chunks for that step via `reduceFileFacts`.\n - Append the consolidated result under the step's name heading.\n\n**4. Component Assembly** \nAfter processing files, append any non-empty child summaries to the components list. If no components exist at all (no files AND no children), clear the module cache and return empty string.\n\n**5. Final Synthesis \u0026 Persistence** \n- Pass all assembled components to `reduceInChunks` for the directory path.\n- Store the result in both the memory cache (`cache.Modules[node.Path]`) and on disk at `cfg.DocsDir/modules/\u003csafe-filename\u003e`.\n- Return the final summary string.\n\n---\n\n### Domain Concepts\n\n| Concept | Description |\n|---|---|\n| **Hierarchical module synthesis** | A directory tree is synthesized bottom-up: files contribute facts, subdirectories contribute their own summaries, and the parent node combines both into a single unified representation. |\n| **Context-aware truncation** | File content limits are not fixedβ€”they scale with `c.NumCtx`, reserving 75% of tokens for file data and 25% for prompts/output. This adapts to different LLM context windows. |\n| **Two-level caching** | Files cache individual facts keyed by SHA256 (content-invariant), while modules cache full synthesized summaries keyed by path. Reuse is possible only when the node is not affected. |\n| **Multi-step extraction pipeline** | Each file passes through a configurable sequence of LLM steps (`cfg.ExtractionSteps`), where each step produces an independent fact set that gets consolidated before being combined with other files' results. |\n| **Affected-directory gating** | The `affectedDirs` map acts as a coarse filter: directories not in this set are skipped entirely if they have cached summaries, avoiding redundant synthesis across unrelated parts of the tree. |\n\n#### [STATE_AND_CONCURRENCY]\n## Mutable State\n\n| Variable | Type | Mutation? | Notes |\n|---|---|---|---|\n| `cache.Files` | map entry (`FileCacheEntry{SHA256: ..., Facts: ...}`) | **Yes** β€” assigned on cache miss after extraction | Local to the recursive call; written under no lock |\n| `cache.Modules[node.Path]` | string (module summary) | **Yes** β€” set at multiple points in the recursion and in child calls | Local to the recursive call; written under no lock |\n\nOther parameters (`node *DirNode`, `cfg *config.Config`) are pointers, but this function only reads their fields β€” it does not mutate them.\n\n## Concurrency Mechanisms\n\n- **None** β€” No `sync.Mutex`, channel, atomic operations, or other locking primitives appear in this file.\n- The recursion is itself a concurrency primitive (caller may invoke from many goroutines), so the map writes to `cache.Files` and `cache.Modules` are unprotected against concurrent access if called with multiple goroutines.\n\n## Summary\n\nMutable state exists (`cache.Files`, `cache.Modules`). No explicit concurrency protection is present in this file; any concurrent callers share risk on those maps.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects (External Communication)\n\n### Network / LLM Calls\n- **`c.CallLLM(ctx, systemPrompt, messages, false)`** β€” invoked once per chunk within `cfg.ExtractionSteps`. Constructs a user message with the file content and steps from all extraction stages, then sends it to an external LLM. The response is stripped of markdown fences and concatenated into the facts string for that step.\n\n### Disk I/O\n- **`tools.ReadFileSafely(repoRoot, f)`** β€” reads each leaf file under `node.Files`. Called only on cache miss (and when no precalculated hash was provided). Swallowed errors here; see error handling below.\n- **`tools.WriteFileSafely(repoRoot, modulePath, []byte(finalSum))`** β€” writes the synthesized final summary to disk at `\u003crepoRoot\u003e/\u003cdocsDir\u003e/modules/\u003csafe-filename\u003e`. Errors are wrapped with context (`fmt.Errorf(\"failed to write module documentation for %s: %w\", node.Path, err)`).\n\n### In-memory Cache (internal state)\n- Reads/writes `cache.Modules[node.Path]` and `cache.Files[f]`. These persist across calls but do not communicate outside the process. SHA256 hashes are computed locally via `sha256.Sum256` + hex encoding; no external hash store is touched.\n\n### Logging (internal events)\n- **`logEvent(\"status\", ...)`** β€” called for cache hits, file-read failures, chunk extraction status messages, and directory synthesis completion. These are informational side effects that do not affect program state or error flow.\n\n---\n\n## Error Handling\n\n### Context Cancellation\nChecked at the top of `synthesizeNode` and again inside the per-file loop:\n```go\nif err := ctx.Err(); err != nil { return \"\", err }\n```\nReturns the context error directly β€” no wrapping.\n\n### LLM Call Errors β†’ Wrapped with Context\nThe only place where an external call failure is surfaced:\n```go\nreturn \"\", fmt.Errorf(\"LLM error extracting %s for %s: %w\", step.Name, f, err)\n```\nWraps the underlying `c.CallLLM` error with a descriptive prefix containing the extraction step name and file path. Propagates up to callers of `synthesizeNode`.\n\n### File Read Errors β†’ Swallowed (Logged + Skipped)\nWhen `tools.ReadFileSafely` fails:\n```go\nlogEvent(\"status\", fmt.Sprintf(\"Warning: failed to read file %s: %v\", f, err))\ncontinue\n```\nNo error is returned. The file is skipped entirely β€” no hash, no facts, nothing added to `components`. This is a silent skip; the caller will never know which specific files were missing unless they inspect cache state or log output.\n\n### File Write Errors β†’ Wrapped with Context\nOnly occurs after successful synthesis:\n```go\nreturn \"\", fmt.Errorf(\"failed to write module documentation for %s: %w\", node.Path, err)\n```\nWrapped with the directory path for traceability. Propagates up.\n\n### Recursive Child Synthesis Errors β†’ Propagated As-Is\nChild `synthesizeNode` calls return errors directly when context is cancelled or LLM steps fail. No wrapping here β€” just `return \"\", err`.\n\n### Empty Components Path β†’ Clean Return (No Error)\nWhen no components exist for a directory, returns `(\"\", nil)` and clears the module cache entry:\n```go\ncache.Modules[node.Path] = \"\"\nreturn \"\", nil\n```\n\n### Summary of Error Patterns\n| Source | Pattern | Propagation |\n|---|---|---|\n| Context cancel | Direct return | Upward |\n| LLM call failure | Wrapped with step + file context | Upward |\n| File read failure | Logged; skipped silently | None (swallowed) |\n| Module write failure | Wrapped with dir path | Upward |\n| Recursive child error | Propagated as-is | Upward |\n\n### No Panics Detected\nThe function uses only `return \"\", err` / `return \"\", fmt.Errorf(...)` paths. No `panic()` or unhandled recover blocks are present in this file." - }, - "internal/engine/tree.go": { - "sha256": "da99a35e3ca638d84c842f22940d62250a4d22ebc4950c9424fcb37873d6ea73", - "facts": "#### [API_SIGNATURES]\n## Public Surface Area of `tree.go`\n\n### Exported Structs\n\n| Type | Fields |\n|------|--------|\n| `FileChange` | `Path string`, `Status string` |\n| `DirNode` | `Path string`, `Files []string`, `Children map[string]*DirNode` |\n\n### Exported Functions\n\n**None.** All functions in this file (`propagateAffected`, `determineAffected`, `buildTree`) use lowercase identifiers and are unexported.\n\n#### [BUSINESS_LOGIC]\n## Domain Concepts\n\n- **File Change Tracking**: `FileChange` models a file operation with path and status (`Added`, `Modified`, `Deleted`). The engine consumes these change events as input.\n\n- **Directory Tree Representation**: `DirNode` is a hierarchical tree node holding files at that level and child directories in its subtree. `buildTree` converts a flat list of file paths into this nested structure, preserving directory nesting by splitting on `/`.\n\n- **Module Mapping**: Each directory is mapped to a module via `ToSafeMarkdownFilename`, which transforms the path into a safe filename for use inside a `modules/` directory. The resulting module path is resolved against `repoRoot` and checked for existence. If the expected module directory does not exist on disk, the node is marked affected.\n\n- **Cached Module Metadata**: A `MetadataCache` stores previously computed module metadata keyed by directory path. If no cached entry exists for a given path (`Modules[n.Path] == \"\"`), the node is treated as newly encountered and marked affected.\n\n## Business Rules\n\n1. **Affected propagation**: When any file in a subtree matches a change, every ancestor directory up to the root is also considered affected. This ensures parent directories are included in downstream processing even if only a leaf changed.\n\n2. **Module existence validation**: A module's directory must exist on disk for it to be valid. Missing directories trigger an \"affected\" flag so that regeneration or rebuild logic can handle them.\n\n3. **Cache-aware detection**: The engine distinguishes between previously cached modules and uncached ones, treating the latter as affected. This avoids re-processing known-good entries while surfacing unknowns.\n\n## Algorithmic Flow\n\n1. Build a directory tree from the provided file list (`buildTree`).\n2. Collect all changed file paths into a lookup map (`determineAffected`).\n3. Traverse the tree recursively:\n - If any direct child of the current node is in the changeset, mark it affected.\n - Compute the module path for this directory and check whether it exists on disk; if not, mark affected.\n - Check the metadata cache; if no cached entry exists, mark affected.\n4. Recurse into each child directory, accumulating affected status upward.\n\n#### [STATE_AND_CONCURRENCY]\n**Mutable State:**\n\n| Variable / Field | Type | Mutated? | Notes |\n|---|---|---|---|\n| `DirNode.Files` | `[]string` | Yes | Appended in `buildTree`, read-only otherwise |\n| `DirNode.Children` | `map[string]*DirNode` | Yes | Populated in `buildTree`; mutated via map assignment |\n| `DirNode.Path` | `string` | No (in this file) | Set at construction only |\n| `affectedDirs` | `map[string]bool` | Yes | Mutated by both `propagateAffected` and `determineAffected` (passed as parameter) |\n| `cache.Modules[n.Path]` | value read from `*MetadataCache` | No (read-only in this file) | Read inside `determineAffected`, not mutated here |\n\n**Concurrency Mechanisms:** None. No `sync.Mutex`, channels, goroutines, or atomic operations are used in this file.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects (I/O)\n\n| Function | Operation | Direction | Detail |\n|----------|-----------|-----------|--------|\n| `determineAffected` | `os.Stat(absModulePath)` | Read (disk) | Stat-check on resolved module path to detect absence |\n| `buildTree`, `propagateAffected` | None | β€” | Pure tree construction / traversal, no I/O |\n\n**Summary**: This file performs exactly one disk-read operation (`os.Stat`). There are **no writes**, **no network calls**, and **no database interactions**. The only external dependency is `security.SafeResolve`, whose behavior cannot be determined without its source.\n\n---\n\n## Error Handling\n\n| Function | Pattern | Detail |\n|----------|---------|--------|\n| `determineAffected` | **Swallowed** | `err == nil` from `security.SafeResolve` β€” non-nil errors are silently dropped (no wrap, no panic). Only the \"not exists\" case is acted on. |\n| All functions | No sentinels, no panics | Errors are not returned to callers; `propagateAffected` and `buildTree` have no error parameters at all. |\n\n**Summary**: The only error path that does something meaningful with errors (`os.IsNotExist`) is a \"file missing\" check β€” a sentinel-like pattern where absence itself is the signal. All other errors are swallowed without propagation or logging." - }, - "internal/engine/utils.go": { - "sha256": "f60c0fac1750c12af2fcd71b42e8ed57d875c2956dfce170384ceca0770a4c0c", - "facts": "#### [API_SIGNATURES]\n# Public Surface Area β€” `internal/engine/utils.go`\n\n## Exported Functions\n\n| Function | Signature |\n|---|---|\n| **`ToSafeMarkdownFilename`** | `func ToSafeMarkdownFilename(modulePath string) string` |\n\n#### [BUSINESS_LOGIC]\n## Analysis: `internal/engine/utils.go`\n\n**What this file does:**\n\nThis is a pure string transformation utility with no business logic, domain concepts, or complex algorithmic flow. It maps a module path to a safe filename for markdown documentation.\n\n**The conversion pipeline:**\n\n1. Replace every `/` character in the input `modulePath` with `_`.\n2. If the result is empty or equals `\".\"`, substitute it with `\"root\"`.\n3. Append `.md` and return.\n\n**Domain concept:** Module-to-document routing β€” given a module identifier, produce a corresponding markdown file path. The function assumes the caller already owns a consistent `modulePath` convention; this utility only adapts that path for filesystem-safe filenames.\n\n#### [STATE_AND_CONCURRENCY]\n**Mutable State:** None. The file contains a single pure function (`ToSafeMarkdownFilename`) that operates on its local parameters only. There are no global variables, structs, or fields that hold changing state.\n\n**Concurrency Mechanisms:** None. No channels, `sync.Mutex`, or other synchronization primitives are used.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects Analysis\n\n**No external I/O.** This code performs zero communication with the outside world:\n\n- **Disk/DB**: None. No file reads, writes, or database operations.\n- **Network**: None. No HTTP calls, DNS lookups, or socket activity.\n- **Stdout/Stderr**: None. No logging, printing, or output to streams.\n- **System-level**: None. No environment variable access, signal handling, or OS calls.\n\nThe only \"effect\" is in-memory string manipulation: replacing `/` with `_`, conditionally substituting `.` or empty strings with `\"root\"`, and appending `.md`. All state resides entirely within the local stack frame.\n\n## Error Handling Analysis\n\n**No error handling.** The function signature returns a plain `string`, not an `error` interface. No panic is raised. No sentinel values, no wrapped errors, no fallback defaults on failureβ€”only default input sanitization:\n\n- If `modulePath` is empty or contains only `/` characters (resulting in `\".\"` or `\"\"` after replacement), it becomes `\"root.md\"`.\n- Otherwise, the path is sanitized and suffixed with `.md`.\n\nThis means **any malformed input** (e.g., a path containing invalid filesystem characters other than `/`, an extremely long string, etc.) passes through untouched. The caller has no way to detect failureβ€”only that an output was produced." - }, - "internal/security/security.go": { - "sha256": "72b9c51b72275e02af44e1a1a64b6cfb525957b4903b3dc1d011202e0503c280", - "facts": "#### [API_SIGNATURES]\n# security.go β€” Public Surface Area\n\n## Package: `security` (internal/security)\n\n---\n\n### Structs\n\n#### `SimpleLock`\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `lockPath` | `string` | Path to the lock file |\n| `file` | `*os.File` | Opened lock file handle |\n| `mu` | `sync.Mutex` | Internal mutex for thread safety |\n| `closed` | `bool` | Internal state tracking whether unlocked/closed |\n\n---\n\n### Functions\n\n#### `SafeResolve(repoRoot, inputPath string) (string, error)`\n\n- **Input:** `repoRoot`, `inputPath` β€” both `string`\n- **Output:** resolved path as `string`, or an `error` if path traversal is detected / stat operations fail.\n\n#### `AcquireLock(repoRoot string) (*SimpleLock, error)`\n\n- **Input:** `repoRoot` β€” `string`\n- **Output:** a `*SimpleLock` pointer and `error`. Returns nil on failure (locked by another process, write error, etc.).\n\n#### `EnsureGitignoreHasLockfile(repoRoot string) error`\n\n- **Input:** `repoRoot` β€” `string`\n- **Output:** `error`, or `nil` if the lock filename is already present in `.gitignore`.\n\n---\n\n### Methods\n\n#### `(l *SimpleLock).Unlock() error`\n\n- **Input:** receiver `*SimpleLock`\n- **Output:** `error`, or `nil` if already closed. Idempotent; safe to call multiple times.\n\n#### [BUSINESS_LOGIC]\n## Business Rules \u0026 Domain Concepts\n\n### 1. Repository Boundary Enforcement\nThe code enforces that all internal paths must resolve strictly within the repository root, preventing path traversal attacks and accidental escapes beyond the project boundary.\n\n### 2. Symlink-Aware Path Resolution\nWhen resolving paths, existing ancestor components are evaluated through symlinks to their physical targets before reconstructing the full resolved path. This prevents traversal via symlinked directories while preserving the logical input structure.\n\n### 3. Process-Level File Locking\nA file-based lock mechanism provides mutual exclusion for concurrent code-reducer processes. The lock is identified by a PID written into a `.code-reducer.lock` file, and it uses `O_EXCL` (exclusive creation) to guarantee atomic acquisitionβ€”no two processes can hold the lock simultaneously on the same filesystem.\n\n### 4. Stale Lock Detection\nIf a process exits without releasing its lock, another process attempting to acquire will detect the stale state and report an error with instructions for manual cleanup. This is intentional behaviorβ€”the system expects the operator to intervene when locks persist across runs.\n\n### 5. Automated Gitignore Integration\nThe lockfile is automatically appended to `.gitignore` so it does not enter version control. If already present, no duplicate entry is written.\n\n## High-Level Algorithmic Flow\n\n```\nβ”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”\nβ”‚ SafeResolve (Path Security) β”‚\nβ”‚ β”‚\nβ”‚ 1. Compute absolute repoRoot β”‚\nβ”‚ 2. Resolve symlinks on the root β”‚\nβ”‚ 3. Walk target path upward until a stat β”‚\nβ”‚ succeeds (finds closest existing ancestor) β”‚\nβ”‚ 4. Rebuild full path from resolved ancestor β”‚\nβ”‚ 5. Verify resolvedPath is inside resolvedRoot β”‚\nβ””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜\n\nβ”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”\nβ”‚ AcquireLock / SimpleLock β”‚\nβ”‚ β”‚\nβ”‚ 1. Resolve lock file path via SafeResolve β”‚\nβ”‚ 2. Attempt atomic creation with O_EXCL β”‚\nβ”‚ - On success: write PID, return lock β”‚\nβ”‚ - On failure (exists): report held β”‚\nβ””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜\n\nβ”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”\nβ”‚ Unlock / EnsureGitignore β”‚\nβ”‚ β”‚\nβ”‚ 1. Close file handle β”‚\nβ”‚ 2. Remove lock file from disk β”‚\nβ”‚ 3. Append lock filename to .gitignore β”‚\nβ”‚ (idempotent: skip if already present) β”‚\nβ””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜\n```\n\n## Summary\n\nThis module solves three domain problems: **security** (preventing path traversal and sandbox escape), **concurrency control** (process-level mutual exclusion via file locks), and **operational hygiene** (keeping lock state out of version control).\n\n#### [STATE_AND_CONCURRENCY]\n## Mutable State \u0026 Concurrency Analysis β€” `internal/security/security.go`\n\n### Mutable State\n\n| Field/Variable | Type | Location | Mutated By | Notes |\n|---|---|---|---|---|\n| `lockPath` | `string` | `SimpleLock` struct field | set once in `AcquireLock()` constructor | Not mutated after construction. Zero value (`\"\"`) if lock is not acquired. |\n| `file` | `*os.File` | `SimpleLock` struct field | set in `AcquireLock()`, set to `nil` in `Unlock()` | Pointer cleared inside the mutex in `Unlock()`. |\n| `mu` | `sync.Mutex` | `SimpleLock` struct field | acquired/released by methods on receiver | Not modified as a value; used for locking. |\n| `closed` | `bool` | `SimpleLock` struct field | set from `false` β†’ `true` inside `Unlock()` (under mutex) | Zero-value-initialized. Once written, stays `true`. |\n\nNo global variables exist in this file. All state is local or encapsulated on `*SimpleLock`.\n\n### Concurrency Mechanisms Protecting Mutable State\n\n| Mechanism | Scope | Protected Fields |\n|---|---|---|\n| `sync.Mutex` (`mu`) | Per-instance, receiver-level | `closed`, `file` (via pointer nil-ing) in `Unlock()` |\n\n**Observations:**\n- The mutex is acquired at the start of `Unlock()` and deferred; it covers all writes to `l.closed` and `l.file`. This serializes Unlock calls on a single lock instance.\n- No other concurrent access paths are visible in this file that touch these fields (e.g., no separate reader or writer path). The `AcquireLock` constructor is not called concurrently against the same receiver, so additional locking there would be unnecessary based on what's shown.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects (I/O Communication)\n\n### Disk / Filesystem Operations\n\n| Function | Read | Write | Delete/Atomic | Notes |\n|---|---|---|---|---|\n| `SafeResolve` | `os.Lstat`, `filepath.EvalSymlinks` (read-only metadata) | None | None | Resolves symlinks on the root, then walks up ancestors until one exists. No actual data read/written. |\n| `AcquireLock` | None | `os.OpenFile` with O_CREATE\\|O_EXCL, then writes PID bytes (`fmt.Sprintf(\"%d\\n\", os.Getpid())`) | None (writes PID) | Creates `.code-reducer.lock` atomically if not present; writes own PID as content. |\n| `SimpleLock.Unlock()` | None | `l.file.Close()` (flushes buffer to disk) | `os.Remove(l.lockPath)` β€” best-effort, non-blocking on error | Idempotent: safe to call twice. |\n| `EnsureGitignoreHasLockfile` | `os.ReadFile(gitignorePath)` | `os.OpenFile(... O_APPEND\\|O_CREATE\\|O_WRONLY ...)` then `f.WriteString(...)` | None | Appends `# Code-Reducer Lockfile\\n.code-reducer.lock\\n` if not already present. Creates `.gitignore` if missing. |\n\n### External Process Communication (Network)\nNone β€” no networking, HTTP, gRPC, or socket usage.\n\n### Database / Remote Storage\nNone β€” no DB calls.\n\n---\n\n## Error Handling Patterns\n\n### Panic Behavior\n**No panics.** All errors are returned as typed `error` values wrapped with context strings.\n\n### Sentinel Errors (Process-Specific)\n\n| Condition | Returned Value | Wrapping Strategy |\n|---|---|---|\n| Lock already held by another process | `lock at %s is already held by another process...` | **Not wrapped** β€” returned as a fresh, standalone error string. Caller gets the actionable message directly. |\n| `os.OpenFile(... O_EXCL)` fails AND NOT `os.IsExist(err)` | `failed to acquire lock at %s: %w` | Wrapped with `%w`. |\n| PID write to lockfile fails | `failed to write pid to lockfile: %w` | Wrapped with `%w`. The file and lock are cleaned up before returning. |\n\n### Non-Sentinel / Generic Errors\n\n| Condition | Returned Value | Wrapping Strategy |\n|---|---|---|\n| `filepath.Abs(repoRoot)` fails | `\"failed to get absolute root path: %w\"` | Wrapped with `%w`. |\n| `filepath.EvalSymlinks(absRoot)` fails | `\"failed to resolve symlinks for root path: %w\"` | Wrapped with `%w`. |\n| `os.Lstat` returns unexpected error (not nil, not IsNotExist) | `\"failed to stat path component: %w\"` | Wrapped with `%w`. |\n| `filepath.EvalSymlinks(current)` fails on ancestor | `\"failed to resolve symlinks for ancestor path: %w\"` | Wrapped with `%w`. |\n| `filepath.Rel` fails OR relative path starts with `..` | `\"security violation: path traversal detected: %q\"` | **Not wrapped** β€” standalone string. Security-critical, so it is unambiguous and not chained. |\n| `.gitignore` read error (not IsNotExist) | `\"error reading .gitignore: %w\"` | Wrapped with `%w`. |\n| Append to `.gitignore` fails | `\"failed to open .gitignore for appending: %w\"` or `\"failed to write to .gitignore: %w\"` | Wrapped with `%w`. |\n\n### Error Wrapping Convention\n- **Standard library errors** (OS, `fmt`, `filepath`) are wrapped via `%w` whenever the caller needs context.\n- **Security-critical violations** (`path traversal detected`) and **actionable process-level messages** (lock held by another) are returned as plain strings β€” no wrapping. This makes them easy to detect with `strings.Contains` or direct comparison if needed.\n- **Swallowed errors**: If `l.file.Close()` inside `Unlock()` fails, the error is discarded (`err = nil`). The lockfile removal happens regardless; any removal failure that isn't `IsNotExist` is preserved in a local variable but only surfaces if close succeeded and remove failed β€” otherwise it's dropped. This is effectively swallowed behavior: best-effort cleanup.\n\n### Lock Acquisition Failure Cleanup\nIn `AcquireLock`, if writing the PID fails, the function calls `f.Close()` then `os.Remove(lockPath)` before returning an error. The file handle and lockfile are cleaned up so no stale partial state remains on disk." - }, - "internal/tools/file_tools.go": { - "sha256": "069bb4a26e37229b88930d5fb26aafd36eeb210ca0f8dbf9248dda29bb750dfe", - "facts": "#### [API_SIGNATURES]\n# Public Surface Area β€” `internal/tools` (`file_tools.go`)\n\n## Functions\n\n| Function | Input | Output |\n|---|---|---|\n| [`ReadFileSafely`](file:///path/to/file_tools.go#L15-L64) | `repoRoot`, `virtualPath` `string` | `[]byte`, `error` |\n| [`WriteFileSafely`](file:///path/to/file_tools.go#L70-L132) | `repoRoot`, `virtualPath` `string`; `content` `[]byte` | `error` |\n| [`LoadGitignore`](file:///path/to/file_tools.go#L138-L164) | `repoRoot` `string` | `[]string`, `error` |\n| [`ShouldIgnoreFile`](file:///path/to/file_tools.go#L170-L236) | `repoRoot`, `relPath` `string`; `gitIgnore` `*ignore.GitIgnore`; `ignoredExtensions` `[]string` | `bool` |\n| [`DiscoverCodeFiles`](file:///path/to/file_tools.go#L242-L291) | `repoRoot` `string`; `ignores`, `ignoredExtensions` `[]string` | `[]string`, `error` |\n| [`IsBinaryFile`](file:///path/to/file_tools.go#L297-L315) | `path` `string` | `bool` |\n\n## Structs / Interfaces / Constants\n\nNone exported.\n\n#### [BUSINESS_LOGIC]\n## Domain Concepts \u0026 Business Rules\n\n### 1. Repository File Discovery\nThe primary business rule is to **recursively walk a repository root and discover high-signal source code files** while filtering out noise: build artifacts, dependency directories, output files, dot-prefixed hidden paths, `.egg-info` markers, binary files, and any user-defined ignore patterns (via gitignore compilation).\n\n### 2. File Ignore Rules (Multi-Layer)\nA file is considered \"ignored\" if it matches **any** of these conditions:\n- Matches a compiled gitignore pattern list\n- Has a path component starting with `.` (dot-prefixed) or ending in `.egg-info`\n- Has an extension matching the user-provided ignored extensions list (case-insensitive, supports both `.` prefix and suffix forms)\n- Is detected as binary\n\nIf any single check returns true, the file is excluded. This is a **union** of ignore conditions.\n\n### 3. TOCTOU Race Condition Mitigation\nBoth read and write operations implement a **Time-Of-Check-Time-Of-Use (TOCTOU) safety pattern**:\n1. Resolve the virtual path to an absolute safe path via `SafeResolve`\n2. Open the file descriptor first\n3. Perform `lstat` on the resolved path to check for symlinks\n4. Perform `fstat` on the open file descriptor\n5. Verify that `lstat` and `fstat` reference the **same inode** β€” if not, a symlink race was detected between opening and checking\n\nThis prevents an attacker from replacing the target at the resolved path with a symlink after resolution but before access verification.\n\n### 4. Read vs Write Safety\n- **Read**: After TOCTOU verification, reads all content and returns it\n- **Write**: After TOCTOU verification, truncates to zero first (only safe because symlinks were already ruled out), then writes the provided content\n\nBoth operations reject any symlink detection outright as a security violation.\n\n### 5. Binary Detection Rule\nA file is considered binary if its first 1024 bytes contain a null byte (`\\x00`). Files that cannot be opened are treated as non-binary (returns false). This avoids loading entire files into memory for classification.\n\n---\n\n## High-Level Algorithmic Flow\n\n```\nDiscoverCodeFiles(repoRoot, ignores, ignoredExtensions)\nβ”œβ”€β”€ Compile gitignore from ignore lines β†’ gitIgnore object\n└── filepath.WalkDir(repoRoot):\n β”œβ”€β”€ Skip repo root itself (first entry)\n β”œβ”€β”€ If directory:\n β”‚ β”œβ”€β”€ Check name against dot-prefix / .egg-info / gitIgnore\n β”‚ └── If any match β†’ Skip entire subtree\n └── If file:\n └── ShouldIgnoreFile check:\n β”œβ”€β”€ gitignore matches? β†’ ignore\n β”œβ”€β”€ path component starts with \".\" or ends \".egg-info\"? β†’ ignore\n β”œβ”€β”€ extension in ignoredExtensions list? β†’ ignore\n β”œβ”€β”€ known text extension (go, js, py, etc.)? β†’ keep\n β”œβ”€β”€ binary? β†’ ignore\n └── otherwise β†’ add to result list\n```\n\n**Result**: A flat list of relative paths representing source code files that pass all safety and filtering rules.\n\n#### [STATE_AND_CONCURRENCY]\n**Mutable State: None.**\n\nNo package-level/global variables exist in this file, and every function operates purely on its parameters with local-only state (local variables, struct fields passed by value or interface). The `gitIgnore *ignore.GitIgnore` parameter is read but never mutated; `files []string` in `DiscoverCodeFiles` is a fresh allocation.\n\n**Concurrency Mechanisms: None.**\n\nNo `sync.Mutex`, channels, atomic operations, or other concurrency primitives are used. All functions are goroutine-safe by construction (no shared mutable state).\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects \u0026 Error Handling Analysis β€” `internal/tools/file_tools.go`\n\n---\n\n### Module-Level Imports Used for I/O\n\n| Package | Purpose in this file |\n|---|---|\n| `os` | File/directory operations (Open, Lstat, MkdirAll, WalkDir, etc.) |\n| `io` | Buffered reading (`io.ReadAll`, `io.EOF`) |\n| `bufio` | Line-scanning the `.gitignore` file |\n| `fmt` | Error wrapping with `%w` and formatted messages |\n| `path/filepath` | Path resolution, walking, joining, splitting |\n| `strings` | Text trimming, prefix/suffix checks, case normalization |\n\n---\n\n### Per-Function Analysis\n\n#### 1. `ReadFileSafely(repoRoot, virtualPath string) ([]byte, error)`\n\n**I/O operations:**\n- Calls `security.SafeResolve()` β€” communicates with the filesystem to resolve a canonical path (the exact behavior depends on that external package; not in this file).\n- Calls `os.Open(safePath)` β€” opens the resolved file.\n- Calls `os.Lstat(safePath)` β€” stat without following symlinks.\n- Calls `f.Stat()` (via deferred close) β€” stat through the open fd, which *does* follow symlinks on most platforms.\n- Calls `io.ReadAll(f)` β€” reads the entire file into memory.\n\n**Error handling:**\n- Errors from `SafeResolve` are **returned as-is**.\n- Errors from `os.Open`, `os.Lstat`, `f.Stat`, and `io.ReadAll` are **wrapped with a prefix string** via `fmt.Errorf(..., err)` using `%w`. This preserves unwrapping for callers.\n- The symlink check (`ModeSymlink != 0`) returns an **unwrapped sentinel error**. If the caller wants to distinguish \"real read failure\" from \"security violation\", they can match on the substring `\"symlink\"` or use `errors.Is` if a custom type were used (none is here).\n- No panic.\n\n---\n\n#### 2. `WriteFileSafely(repoRoot, virtualPath string, content []byte) error`\n\n**I/O operations:**\n- Calls `security.SafeResolve()` β€” same external dependency as above.\n- Calls `filepath.Dir(safePath)` β€” computes parent directory path (no I/O).\n- Calls `os.MkdirAll(dir, 0755)` β€” creates the parent directory if missing.\n- Calls `os.OpenFile(safePath, os.O_WRONLY|os.O_CREATE, 0644)` β€” opens for writing, creating if needed.\n- Calls `os.Lstat(safePath)` and `f.Stat()` β€” same TOCTOU race check as ReadFileSafely.\n- Calls `f.Truncate(0)` β€” truncates to zero length before writing.\n- Calls `f.Write(content)` β€” writes the new content.\n\n**Error handling:**\n- Errors from `SafeResolve`, `os.MkdirAll` are **wrapped with `%w`**.\n- Errors from `OpenFile`, `Lstat`, `Stat`, `Truncate`, and `Write` are all **wrapped with a descriptive prefix + `%w`**.\n- Symlink detection returns an **unwrapped sentinel string** (`\"security violation: symlink detected on write\"`). Unwrap-able only by substring match.\n- No panic, no swallowed errors β€” every operation's error is returned to the caller (except `SafeResolve`, which is passed through).\n\n---\n\n#### 3. `LoadGitignore(repoRoot string) ([]string, error)`\n\n**I/O operations:**\n- Constructs `.gitignore` path via `filepath.Join`.\n- Calls `os.Open(gitignorePath)` β€” reads the file.\n- Scans line-by-line with `bufio.Scanner`.\n\n**Error handling:**\n- If `os.IsNotExist(err)` is true β†’ returns `(nil, nil)`. **Silently treats missing `.gitignore` as success.** This is a deliberate contract: no `.gitignore` means \"no patterns to ignore\".\n- Any other error from `Open` (permission denied, too long path, etc.) is **returned directly** β€” unwrapped. The caller receives the raw `os.*Error`.\n- After scanning, returns `scanner.Err()` if a read error occurred mid-scan. This could be an I/O short-read or scanner overflow; treated as an error return rather than swallowed.\n\n---\n\n#### 4. `ShouldIgnoreFile(repoRoot, relPath string, gitIgnore *ignore.GitIgnore, ignoredExtensions []string) bool`\n\n**I/O operations:**\n- No direct file I/O in this function's body.\n- Calls `security.SafeResolve()` β€” same external dependency as above.\n- Eventually calls `IsBinaryFile(absPath)` at the end of the function if none of the earlier checks matched.\n\n**Error handling:**\n- If `SafeResolve` returns an error β†’ returns `true`. **Treats any filesystem resolution failure as \"ignored\"**. This is a defensive default: anything outside the repo root or with a bad path gets ignored rather than causing a caller-visible error. The error is swallowed and converted to a boolean decision.\n- No panic, no wrapping β€” this function only returns `bool`.\n\n---\n\n#### 5. `DiscoverCodeFiles(repoRoot string, ignores []string, ignoredExtensions []string) ([]string, error)`\n\n**I/O operations:**\n- Calls `ignore.CompileIgnoreLines(ignores...)` β€” in-memory compile of ignore rules (no I/O).\n- Calls `filepath.WalkDir(repoRoot, ...)` β€” walks the entire directory tree. This is a significant filesystem traversal: reads every directory entry and file metadata on disk.\n\n**Error handling:**\n- The walk callback receives an error per entry (`err error`). If non-nil β†’ returns `nil` from the callback (which tells `WalkDir` to skip that entry). **The underlying walk error is swallowed at each node.** This means if a directory cannot be read, entries inside it are silently skipped β€” no error propagates.\n- The final return value is `(files, err)`. Since every internal error in the walk callback returns `nil`, and the function only ever calls `filepath.Rel` (which errors are returned as `nil` from the callback), **the returned `err` will always be `nil`**. The caller has no way to detect filesystem traversal failures β€” they are completely hidden.\n- No panic, no sentinel errors β€” this is a best-effort discovery function with silent error swallowing at every layer.\n\n---\n\n#### 6. `IsBinaryFile(path string) bool`\n\n**I/O operations:**\n- Calls `os.Open(path)` β€” opens the file for reading.\n- Reads up to 1024 bytes into a buffer.\n- Scans for null byte (`buf[i] == 0`).\n\n**Error handling:**\n- If `Open` fails β†’ returns `false`. **Read permission errors, ENOENT, etc., are silently treated as \"not binary\"**. The caller cannot distinguish between \"file is text\" and \"could not open file\".\n- After reading, if the error is *anything other than* `io.EOF`, it also returns `false`. This means:\n - Short read (e.g., partial EOF) β†’ `false`\n - Unexpected I/O error during read β†’ `false`\n - Only `io.EOF` after a successful read loop falls through to the null-byte check.\n- No panic, no wrapping β€” this is a simple best-effort boolean with all errors swallowed into `false`.\n\n---\n\n### Summary Table\n\n| Function | Swallowed Errors? | Wrapped (`%w`) | Sentinel (unwrapped) | Panic |\n|---|---|---|---|---|\n| `ReadFileSafely` | No | Yes, for every I/O error | Symlink race check | No |\n| `WriteFileSafely` | No | Yes, for every I/O error | Symlink race check | No |\n| `LoadGitignore` | Yes (missing file β†’ nil) | No | None | No |\n| `ShouldIgnoreFile` | Yes (`SafeResolve` err β†’ true) | No | None | No |\n| `DiscoverCodeFiles` | Yes, extensively | No | None | No |\n| `IsBinaryFile` | Yes (all open/read errs β†’ false) | No | None | No |\n\n**Key pattern:** The security-critical functions (`ReadFileSafely`, `WriteFileSafely`) are the only ones that wrap errors and return sentinel violations unwrapped. Everything else uses silent failure β€” either returning `nil` for \"not found\" or swallowing into a boolean default. This is intentional: discovery and classification functions should not fail callers; they degrade gracefully." - }, - "internal/tools/git_tools.go": { - "sha256": "ee9d244fdd8ca3a1dfcc7df969b02a4d68d65c2fe3aba8203d62819bed4efcd0", - "facts": "#### [API_SIGNATURES]\n## Public Surface Area: `internal/tools/git_tools.go`\n\n### Exported Functions\n\n| Function | Parameters | Return Type |\n|---|---|---|\n| `RunGit(repoRoot string, args ...string)` | `repoRoot string`, variadic `args ...string` | `(string, error)` |\n| `VerifyGitRepo(repoRoot string)` | `repoRoot string` | `error` |\n\n### Notes\n\n- No exported structs or interfaces are defined in this file.\n- No internal logic is reported; only signatures and types are captured above.\n\n#### [BUSINESS_LOGIC]\n**Domain Concepts:**\n- **Git Process Abstraction**: Encapsulates OS process spawning for Git commands, allowing the application layer to invoke Git without directly managing command execution or stream handling.\n- **Repository Integrity Check**: Validates whether a specific filesystem path belongs to a valid Git working tree before proceeding with repository-dependent operations.\n\n**Business Rules:**\n1. Any Git subcommand can be executed within a designated root directory via the `RunGit` interface, which suppresses paging output and captures both stdout and stderr for analysis.\n2. A directory is considered a valid Git repository only if the `git rev-parse --is-inside-work-tree` command succeeds; any failure indicates the path is outside a Git working tree or not a git repository at all.\n\n**Algorithmic Flow:**\n1. **Execution Phase**: Initialize a process running `git --no-pager` with user-defined arguments, restricted to the provided root directory. Capture output streams into buffers and execute. If execution succeeds, return trimmed stdout; if it fails, return a structured error containing stderr content and exit information.\n2. **Verification Phase**: Delegate to the execution phase with specific arguments (`rev-parse`, `--is-inside-work-tree`). Interpret success as confirmation of repository validity. Interpret failure as an assertion that the directory is not a Git repository.\n\n#### [STATE_AND_CONCURRENCY]\n**Mutable State:** None. There is no global variable, no struct with changing fields, and no function parameters that persist across calls. All local variables (`cmd`, `stdout`, `stderr`) are created fresh on every invocation and discarded afterward.\n\n**Concurrency Mechanisms:** None. No `sync.Mutex`, channels, or other concurrency primitives are present.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects (External Communication)\n\n| Channel | Mechanism | Scope |\n|---|---|---|\n| **Process Execution** | Spawns `git` binary via `exec.Command(\"git\", ...)` β€” both functions call it synchronously. | Local filesystem + optional network (git may fetch/push remotes during execution). |\n| **File System Indirect Access** | Running git commands causes read/write on `.git/`, `HEAD`, refs, work tree, index, etc. | Only the repo at `repoRoot`. No direct disk I/O by this package itself. |\n| **Network (transient)** | Not initiated by this code; it depends entirely on what the spawned git process does. If no network calls happen during execution, none occur here. |\n\n**No standard library packages are imported beyond `bytes`, `fmt`, `os/exec`, `strings`.** No DB drivers, HTTP clients, or other I/O libraries are used by this module.\n\n---\n\n## Error Handling / Returns\n\n| Function | Failure Path | Behavior |\n|---|---|---|\n| **`RunGit`** | `cmd.Run()` returns non-nil error (exit code β‰  0). | Wraps the error: `fmt.Errorf(\"git command failed: %v, stderr: %s\", err, trimmedErr)`. Returns `(string, error)` with stdout and wrapped error. No panic. |\n| **`RunGit`** | `cmd.Run()` returns nil (exit code 0). | Returns `(trimmedOut, nil)`. Stdout is whitespace-trimmed; stderr is discarded on success. |\n| **`VerifyGitRepo`** | `RunGit(...)` returns error. | Wraps it: `fmt.Errorf(\"not a git repository (or any of the parent directories)\")`. The original stderr content is lost in this wrapper. Returns only the wrapped sentinel-like error. No panic. |\n\n**Error shape:** All errors are `*errors.Error` via `fmt.Errorf` β€” no custom sentinel types, no wrapping with `errors.Join`, no structured codes. Callers receive a single string error describing what happened." - }, - "main.go": { - "sha256": "16ab362b1830374f0306a1990dba8d51da887c1ad126f311c03d242933d2a516", - "facts": "#### [API_SIGNATURES]\n## Exported Public Surface Area of `main.go`\n\n### Structs\n*None defined.*\n\n### Interfaces\n*None defined.*\n\n### Methods\n*None defined.*\n\n### Functions\n\n- **main** (package-level)\n - Input: None\n - Output: None\n\n#### [BUSINESS_LOGIC]\n## Analysis of main.go\n\n### What This File Solves\n\nThis file implements **zero business logic or domain concepts**. It is purely an entry point / bootstrap for a CLI application.\n\n### High-Level Algorithmic Flow\n\n1. Invoke `cmd.RootCmd.Execute()` β€” delegates all command routing and execution to the external `github.com/arrase/code-reducer/cmd` package\n2. If execution fails, print the error to stderr and exit with code 1\n\nThat is it. All business rules, parsing, subcommands, validation β€” those live in the imported `cmd` package, not here.\n\n#### [STATE_AND_CONCURRENCY]\n**Mutable State:** No mutable state. The `main()` function contains no variable declarations and no field modifications. All logic is delegated to the external package `cmd.RootCmd.Execute()`.\n\n**Concurrency Mechanisms:** No concurrency mechanisms are used in this file.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n**Side Effects \u0026 I/O Communication**\n\n* **Stderr Output**: Writes a formatted string (\"Execution failed: \") followed by the raw error message to `os.Stderr` via `fmt.Fprintf`.\n* **Process Termination**: Calls `os.Exit(1)`, which immediately terminates the process with exit code 1, signaling failure to the parent shell or OS.\n* **External Dependency Call**: Invokes `cmd.RootCmd.Execute()` from package `github.com/arrase/code-reducer/cmd`. The behavior of this call is not observable within this file; it returns an error interface which is then checked.\n\n**Error Handling Strategy**\n\n* **Nil Check on Return Value**: Checks if the return value from `Execute()` is non-nil (`if err := ... ; err != nil`).\n* **No Error Wrapping**: The error is passed directly to `%v` for printing; it is not wrapped in a new error type.\n* **No Panic Usage**: Does not use `panic`; exits cleanly via status code 1 instead.\n* **No Sentinel Types**: Uses the standard `error` interface returned by the dependency, rather than a custom sentinel type defined locally." - } - }, - "modules": { - ".": "# Code-Reducer β€” Architecture Documentation\n\n---\n\n## Module Responsibility \u0026 Data Flow\n\nCode-Reducer is a terminal-based documentation-generation agent that produces and maintains technical wikis from Go source code by invoking an Ollama-compatible LLM endpoint. The module implements three phases: **initialization** (repository scan + wiki scaffolding), **interactive setup** (configuration prompts β†’ `.code-reducer.yaml` persistence), and **incremental update** (change detection β†’ selective regeneration). All pipeline execution routes through a single `cmd.RootCmd.Execute()` entry point, with subcommand handlers delegating to an external executor.\n\n---\n\n## Package: cmd β€” CLI Command Wiring \u0026 Lifecycle Management\n\n### Root Command Initialization (`root.go`)\n\nThe root command registers two persistent flags via `PersistentFlags`: `--model-id` (bound to package-level `modelIdFlag`) and `--num-ctx` (bound to package-level `numCtxFlag`). On invocation, the following sequence executes:\n\n1. **Environment validation** β€” `os.Getwd()` resolves the current working directory; failure is wrapped as `\"failed to get current working directory: %w\"` and returned immediately.\n2. **Configuration resolution** β€” merges CLI flags, environment variables, and an optional persistent config file loaded via `internal/config`. Resolution details are delegated outside this package boundary.\n3. **Implicit setup on first run** β€” if the persistent config file does not exist *and* stdin is a terminal (`isatty.IsTerminal(os.Stdin.Fd())` / `isatty.IsCygwinTerminal(os.Stdin.Fd())`), an automatic initial configuration step triggers without requiring user intervention. On non-TTY (CI / piping), this step is skipped and the caller must configure manually via `setup`.\n4. **Mode-gated lifecycle checks** β€” enforced only within the runner/engine layer; no validation of project state occurs at registration time. The two modes (`init`, `update`) are mutually exclusive on an already-initialized project; enforcement lives in the external executor.\n5. **Graceful shutdown** β€” installs SIGINT / SIGTERM handlers via `signal.NotifyContext`. A deferred `stop()` cancels the context; any in-flight work that respects the cancelled context will exit. No error-return path exists for signal handling within this file.\n6. **Execution loop** β€” delegates to `engine.NewRunner.Run()` with the resolved config and current directory. The runner emits typed events (`status`, `error`, default).\n\nCobra output suppression: `RootCmd` has `SilenceErrors: true` and `SilenceUsage: true`. Errors that would otherwise be printed by cobra's default handler are suppressed β€” callers must handle errors explicitly. No panic calls observed; all failure paths return errors propagated to cobra.\n\n### Interactive Configuration Setup Flow (`setup.go`)\n\nThe `RunSetupFlow(repoRoot)` function generates a `.code-reducer.yaml` configuration file through an interactive prompt loop:\n\n1. **Resolve existing state** β€” attempt to load any previously saved config from the current working directory via `config.LoadConfig(repoRoot)`. If none exists, all prompts default to built-in constants (Ollama defaults for model ID, base URL, context size; `\"wiki\"` for docs directory).\n2. **Prompt user iteratively** for each setting: LLM Model ID β†’ string, Ollama Base URL β†’ string, Ollama Context Size β†’ integer (parsed via `strconv.Atoi`), custom directories/files to ignore β†’ comma-separated list, file extensions to ignore β†’ comma-separated list, documentation directory β†’ string.\n3. **Preserve existing values on empty input** β€” if the user presses Enter without typing anything, the previously loaded (or default) value is kept.\n4. **Support reset semantics** β€” for list-type inputs, the literal strings `\"clear\"` or `\"none\"` wipe the list entirely; otherwise comma-splitting produces the new list.\n5. **Error swallowing on read failures** β€” if stdin reading fails during any prompt (`bufio.NewReader(os.Stdin)` + `reader.ReadString('\\n')`), the error is logged to stderr and execution continues using existing values (no rollback, no restart).\n6. **Persist final config** β€” after all prompts complete, a single `config.Config` struct is constructed from user input + preserved state and written to disk via `config.SaveConfig`. The success message references a `.yaml` file path defined in the config package.\n\nNon-destructive defaults: existing configuration is always preferred over built-in constants; new values only replace what the user explicitly provided (or left as default). Idempotent re-run: running `setup` again when a config already exists produces an incremental update rather than a full overwrite. No retry logic β€” `SaveConfig` is called once; if it fails, the entire flow aborts.\n\n### Update Subcommand Registration (`update.go`)\n\nThis file exports no public functions, structs, or interfaces. The handler registers the CLI entry point via `init()` (package-level) and delegates to `executeCommand(\"update\")`, which resolves the actual update logic outside this package boundary. All substantive logic is deferred to the external function; this file serves only as registration/wiring within the cobra command hierarchy.\n\n### Init Subcommand Registration (`init.go`)\n\nThis file exports no public functions, structs, or interfaces. The handler registers the CLI entry point via `init()` (package-level) and delegates to `executeCommand(\"init\")`, which performs the repository scan and wiki page generation outside this package boundary. All substantive logic is deferred to the external function; this file serves only as registration/wiring within the cobra command hierarchy.\n\n---\n\n## Package: internal/config β€” Multi-Layer Configuration Resolution\n\n### Responsibility\n\nImplements a four-tier precedence pipeline for resolving all configurable values. Each layer overwrites prior values when present; missing layers are transparently skipped. Three properties are resolved independently (`ModelID`, `OllamaBaseURL`, `OllamaNumCtx`). The package also structures the analysis pipeline into named **extraction steps** (`ExtractionStep`), each carrying a distinct prompt template. All non-structural state (resolved config, local variables) is ephemeral β€” scoped to function calls and returned as values. No package-level mutable state survives beyond initialization. Concurrency primitives are absent; the package is single-threaded in design.\n\n### Types\n\n#### `Config`\n\n```go\ntype Config struct {\n ModelID string // resolved model identifier (last-wins: default β†’ YAML β†’ env β†’ flag)\n OllamaBaseURL string // resolved base URL (default β†’ YAML β†’ env; CLI flag omitted)\n OllamaNumCtx int // resolved context size, validated positive (default β†’ YAML β†’ env β†’ flag)\n DocsDir string // path to the documents directory under analysis\n ExtractionSteps []ExtractionStep // ordered list of extraction phases\n Ignore []string // absolute paths excluded from traversal\n IgnoreExtensions []string // file extensions excluded from traversal\n}\n```\n\n#### `ExtractionStep`\n\n```go\ntype ExtractionStep struct {\n Name string // step identifier (e.g., \"api-signatures\", \"business-logic\")\n Prompt string // prompt template applied to this phase during analysis\n}\n```\n\nInstances are held in a package-level slice (`DefaultExtractionSteps`). The slice is immutable after declaration; callers receive copies or slices of it.\n\n### Constants and Defaults\n\n| Constant | Value | Purpose |\n|---|---|---|\n| `CodeReducerModelIdEnvKey` | `\"CODE_REDUCER_MODEL_ID\"` | Overrides `Config.ModelID` when set. |\n| `OllamaBaseUrlEnvKey` | `\"OLLAMA_BASE_URL\"` | Overrides `Config.OllamaBaseURL` when set. |\n| `OllamaNumCtxEnvKey` | `\"OLLAMA_NUM_CTX\"` | Overrides `Config.OllamaNumCtx`; value must parse as a positive integer. |\n| `OllamaDefaultBaseURL` | `\"http://localhost:11434\"` | Fallback base URL when no prior layer supplies one. |\n| `OllamaDefaultModelID` | `\"ornith:9b\"` | Default model identifier. |\n| `OllamaDefaultNumCtx` | `8192` | Default context size. |\n| `ConfigFileName` | `.code-reducer.yaml` | Filename resolved relative to the current working directory. |\n\nThree slices are declared at package init and never mutated:\n\n```go\nvar DefaultIgnores []string // system-supplied ignore paths\nvar DefaultIgnoredExtensions []string // system-supplied ignored extensions\nvar DefaultExtractionSteps []ExtractionStep // ordered extraction phases\n```\n\n### Public API Surface\n\n#### Path Resolution and Existence Check\n\n##### `getConfigPath(cwd string) (string, error)`\n\nConstructs the full path to `.code-reducer.yaml` by joining `cwd` with `ConfigFileName`. Returns a non-nil error only on I/O failure during construction; normal cases return `nil`.\n\n##### `ConfigExists(cwd string) bool`\n\nReturns whether the configuration file exists at the resolved path. Internally calls `os.Stat`; all errors (permission denied, I/O failures, etc.) are swallowed and treated as \"does not exist\". The function returns no error.\n\n#### Configuration Loading\n\n##### `LoadConfig(cwd string) (*Config, error)`\n\nReads and parses the YAML configuration file at the resolved path. Error handling:\n- File-read failure (`os.ReadFile`) β†’ returned to caller unchanged (pass-through).\n- YAML parse failure β†’ wrapped as `\"failed to parse yaml config: %w\"`.\n\nReturns a non-nil `*Config` on success; the struct is initialized with zero values before parsing, so callers receive a populated object even when the file contains only valid empty mappings.\n\n##### `ResolveConfig(repoRoot, modelIdFlag, numCtxFlag string) (*Config, error)`\n\nTop-level entry point for full configuration resolution:\n1. Calls `LoadConfig` to read the YAML file. If the error is a non-existent-file sentinel (`os.IsNotExist(err)`), it swallows and initializes an empty `Config{}`; all other errors propagate wrapped as `\"failed to load configuration file:\"`.\n2. Layers defaults (hard-coded constants) over the loaded config.\n3. Layers environment variable overrides using `os.Getenv` for `CODE_REDUCER_MODEL_ID`, `OLLAMA_BASE_URL`, and `OLLAMA_NUM_CTX`. Non-empty values are used; empty strings fall through. Env read errors are swallowed.\n4. Applies CLI flags (`modelIdFlag`, `numCtxFlag`). The flag value is parsed via `strconv.Atoi`; only `err == nil \u0026\u0026 n \u003e 0` is accepted β€” invalid or non-positive values cause the prior layer's value to be retained silently.\n\nReturns a fully resolved `*Config`. Errors are either pass-through (from `LoadConfig`) or wrapped with `\"failed to load configuration file:\"`. Notable: if YAML is malformed, the parse error wraps and propagates; otherwise resolution always succeeds for missing files.\n\n#### Configuration Persistence\n\n##### `SaveConfig(cwd string, cfg *Config) error`\n\nSerializes the config struct back to disk using `os.WriteFile` with `0600` permissions. Error handling:\n- Write success β†’ returns `nil`.\n- Parse failure (marshal YAML) β†’ wrapped as `\"failed to marshal yaml: %w\"`.\n- File-write failure β†’ wrapped as `\"failed to write config file: %w\"`.\n\nWrites directly (no temp-file + rename); a partial write can leave a truncated file on disk. Permission normalization is implicit β€” existing files with different permissions will fail the write.\n\n---\n\n## Package: internal/engine β€” Code Documentation Pipeline Engine\n\n### Responsibility\n\nImplements a Map-Reduce pipeline that generates and maintains technical documentation for a Go codebase using Ollama-compatible LLM inference endpoints. The engine produces three artifacts per run: `architecture.md` (global overview), `quickstart.md` (onboarding guide), and per-module API summaries under `modules/`. Execution is gated by a repository-level lock to prevent concurrent runs against the same root, and state persistence across invocations is handled by an on-disk metadata cache.\n\n### Pipeline Orchestration\n\n#### Runner Entry Point (`runner.go` β€” `Runner.Run`)\n\nThe public entry point for all pipeline invocations. Responsibilities: repository isolation, concurrency guard acquisition, LLM client construction from config, and mode-gated dispatch to the appropriate pipeline method.\n\n```\nRun(ctx, repoRoot, mode string, onEvent func(Event)) error\n β”œβ”€ ensure .gitignore lockfile entry (best-effort; failure logged as warning)\n β”œβ”€ acquire repository-level lock via security.AcquireLock(repoRoot)\n β”‚ └─ defer lock.Unlock() β€” released on any return path including errors\n β”œβ”€ build *LLMClient from r.cfg.ModelID, BaseURL, NumCtx\n β”œβ”€ switch mode:\n β”‚ case \"init\": client.RunInit(ctx, repoRoot, r.cfg, onEvent)\n β”‚ case \"update\": client.RunUpdate(ctx, repoRoot, r.cfg, onEvent)\n β”‚ default: return fmt.Errorf(\"unsupported mode: %s\", mode)\n```\n\nThe `cfg *config.Config` field is read-only within this file; all mutations to the runner's configuration originate before construction. The lock returned by `security.AcquireLock(repoRoot)` is a local variable with deferred cleanup β€” its internal type and mutual exclusion semantics are not observable from this source alone.\n\n#### Orchestrator Pipelines (`orchestrator.go` β€” `RunInit`, `RunUpdate`)\n\n##### Full Run (`RunInit`)\n\nExecutes an all-or-nothing regeneration regardless of cache state:\n1. **Setup** β€” Load `.gitignore` via `tools.LoadGitignore(repoRoot)`; merge with user ignore list; load metadata cache from disk via `cache.loadMetadataCache()`. Failures logged as warnings, not returned.\n2. **Map Phase** β€” Discover code files via `tools.DiscoverCodeFiles(...)`, compute SHA-256 per file via `computeSHA256(repoRoot, f)`, build directory tree via `buildTree(codeFiles)`. Hash computation errors silently skipped in loops; only stored if hash computed successfully.\n3. **Mark All Affected** β€” Recursively flag every node as affected (init starts fresh).\n4. **Reduce Phase** β€” Call `synthesizeNode` bottom-up on the tree to produce a root summary string and per-module summaries via LLM calls through `c.CallLLM`.\n5. **Generate Standard Docs** β€” Produce `architecture.md` and `quickstart.md` from the root summary via two additional `CallLLM` invocations: one for architecture, one for quickstart. Writes via `tools.WriteFileSafely`; errors wrapped with context (e.g., `\"failed to write quickstart.md: %w\"`).\n6. **Update AGENTS.md** β€” Read existing file via `tools.ReadFileSafely(repoRoot, \"AGENTS.md\")`. If missing, create with guidelines header. If present but does not contain \"AI Agent Guidelines\", append separator + new content block. Only writes when content actually differs. Errors: if read fails but write succeeds β†’ `\"failed to write AGENTS.md: %w\"`; if both fail β†’ wrapped errors chain.\n7. **Teardown** β€” Persist metadata cache via `saveMetadataCache(repoRoot, docsDir, cache)`. Failure logged as warning, function always returns nil at end regardless of this failure.\n\n##### Incremental Run (`RunUpdate`)\n\nExecutes only re-generations for directories whose contents actually changed:\n1. **Setup** β€” Load pipeline state from metadata cache (same path as init).\n2. **Map Phase** β€” Discover code files, hash each via `computeSHA256`, build tree.\n3. **Change Detection** β€” Compare current file hashes against cached hashes in `cache.Files`:\n - New file in codebase but absent from cache β†’ `FileChange{Status: \"Added\"}`\n - Existing file with different hash β†’ `FileChange{Status: \"Modified\"}`\n - Cached file no longer in codebase β†’ `FileChange{Status: \"Deleted\"}` (also removed from cache)\n4. **Prune Stale Modules** β€” Remove any module summaries whose directory is no longer active, delete their markdown files on disk via `os.Remove(absModuleFile)` (error assigned to `_`, never logged).\n5. **Propagate Affected** β€” Run tree-aware propagation (`propagateAffected`/`determineAffected`) to mark directories affected by changes. Errors from these calls are discarded without capture.\n6. **Early Exit** β€” If zero directories are affected β†’ pipeline ends with \"up to date\" status, no further synthesis occurs.\n7. **Conditional Reduce** β€” If root-level change detected or global docs don't exist on disk (`os.Stat(absArch)`/`os.Stat(absQs)`), run full synthesis; otherwise skip and reuse existing `architecture.md`/`quickstart.md`.\n8. **Teardown** β€” Save updated cache via `saveMetadataCache(repoRoot, docsDir, cache)`.\n\n### Directory Tree Construction \u0026 Affected Propagation (`tree.go`)\n\n#### Type Definitions\n\n| Type | Fields | Purpose |\n|------|--------|---------|\n| `FileChange` | `Path string`, `Status string` | Models a file operation with path and status (`Added`, `Modified`, `Deleted`). Consumed by `determineAffected`. |\n| `DirNode` | `Path string`, `Files []string`, `Children map[string]*DirNode` | Hierarchical tree node holding files at that level and child directories in its subtree. Produced by `buildTree`. |\n\n#### Functions (all unexported)\n\n- **`buildTree(codeFiles)`** β€” Converts a flat list of file paths into nested `*DirNode` structure, preserving directory nesting by splitting on `/`. Appends to `DirNode.Files`; populates `DirNode.Children` via map assignment.\n- **`propagateAffected(tree, changeset)`** β€” Traverses the tree recursively: if any direct child of the current node is in the changeset, mark it affected. Accumulates affected status upward through ancestors. Returns mutated `affectedDirs`.\n- **`determineAffected(tree, changeset)`** β€” Computes module path for each directory via `ToSafeMarkdownFilename`, checks existence on disk via `os.Stat(absModulePath)`, reads metadata cache (`cache.Modules[n.Path] == \"\"`), marks node affected if any check fails. Only `os.IsNotExist` is acted on; all other errors are silently dropped.\n\n### LLM Client Abstraction \u0026 Response Parsing\n\n#### Type: `LLMClient`\n\n```go\ntype LLMMClient struct {\n ModelID string\n BaseURL string\n NumCtx int\n HTTPClient *http.Client // default transport, no custom config (no proxy/redirect/TLS)\n}\n```\n\nConstructed via `NewLLMClient(modelID, baseURL, numCtx)` with a 10-minute HTTP timeout. Fields set once; never reassigned within any method.\n\n#### Methods on `*LLMClient`\n\n##### `CallLLM(ctx context.Context, systemPrompt string, messages []Message, jsonFormat bool) (string, error)`\n\nFull lifecycle: prepends system prompt as first message with role `\"system\"` β†’ serializes into Ollama `/api/chat` JSON payload with model ID and context window size β†’ non-streaming HTTP POST β†’ response body fully read on 200 OK or truncated at 1 KB via `io.LimitReader(resp.Body, 1024)` on non-OK status. Returns only assistant content string; returns empty string on failure.\n\n**Error behavior:**\n\n| Failure path | Error type | Wrapping strategy |\n|---|---|---|\n| JSON marshal fails in request prep | `fmt.Errorf(\"failed to marshal request: %w\", err)` | Wrapped with `%w` |\n| `http.NewRequestWithContext` fails | raw error from `NewRequestWithContext` | **Not wrapped** β€” passed through as-is |\n| HTTP Do (network-level failure) | raw error from `c.HTTPClient.Do(req)` | **Not wrapped** |\n| Body read fails on 200 OK path | raw error from `io.ReadAll` | **Not wrapped** |\n| JSON unmarshal fails on response | `fmt.Errorf(\"failed to parse response: %w\", err)` | Wrapped with `%w` |\n| Non-OK status (body truncated at 1 KB) | `fmt.Errorf(\"ollama api error: status %d, response: %s\", ...)` | New error constructed; body discarded if read fails |\n\n##### `GetBaseSystemPrompt() string`\n\nReturns the fixed base role definition and defensive guidelines inherited by every LLM call regardless of task type.\n\n##### `GetDefaultSystemPrompt(command string) string`\n\nAppends command-specific instructions based on the command string: two named variants exist (`module_synthesis`, `architecture`); any other value receives only the base layer.\n\n### Response Parsing Layer (`json_parser.go`)\n\nInternal helper `extractBalancedJSON` is lowercase and excluded from public surface area.\n\n#### Business Rules\n\n1. Markdown code fences (` ``` `) surrounding JSON must be stripped before parsing β€” handled by `StripOuterMarkdownFence`.\n2. Balanced bracket matching required β€” strings containing `{`, `[`, `}`, `]` literals (including escaped characters like `\\n`) must not break extraction logic β€” handled by internal rune-level scanner with escape state tracking inside string literals.\n3. When multiple balanced JSON candidates exist in a single response, each must be attempted individually until one succeeds, or the entire trimmed string is tried as a final fallback β€” handled by `CleanJSONResponse`/`UnmarshalJSONResponse`.\n\n#### Functions\n\n| Function | Behavior | Error surface | Swallowed? |\n|---|---|---|---|\n| `StripOuterMarkdownFence(s string) string` | Strips surrounding markdown fences; falls back to trimmed input if none match | None β€” returns string only | N/A (no error return) |\n| `CleanJSONResponse(s string) string` | Iterates stripped content, calls `extractBalancedJSON`, returns first successful candidate or original trimmed input unchanged | None β€” returns string only | Yes β€” unbalanced JSON inside response is swallowed silently |\n| `UnmarshalJSONResponse(s string, target interface{}) error` | Multi-layered attempt: (1) collect balanced candidates, try each into target; keep last error in `lastErr`; (2) if any succeeded, return nil; otherwise try entire trimmed string; (3) terminal errors β€” `\"failed to unmarshal JSON candidates: %w\"` or `\"no valid JSON found in response\"` | Explicit (`error`) | Partial parse failures collected and re-emitted via `%w` on first path; \"no valid JSON\" covers second path |\n\n### Synthesis Engine \u0026 Chunking (`chunking.go`, `synthesize.go`)\n\n#### Context-Aware Chunking (all unexported)\n\nThe system processes extracted code/fact items in batches constrained by LLM context limits (`NumCtx * 3`). Three distinct processing modes exist:\n\n##### Module Synthesis (`reduceInChunks`)\n\nSynthesizes architectural-level nodes into architecture descriptions using a `\"module_synthesis\"` system prompt. Each batch represents evidence contributing to understanding the module's design intent. Response passes through `StripOuterMarkdownFence`. Errors wrapped with `\"LLM error during synthesis: %w\"`.\n\n##### File Fact Consolidation (`reduceFileFacts`)\n\nConsolidates extracted facts about a specific step within a file with deduplication and merging. System prompt built locally by concatenating `c.GetBaseSystemPrompt()` + fixed role string. Response also passes through `StripOuterMarkdownFence`. Errors wrapped with `\"LLM error during file fact consolidation: %w\"`.\n\n##### Recursive Convergence (`reduceItems`)\n\nCore business rule: reduce items in batches β†’ collect intermediate results β†’ recursively apply reduction until everything converges into a single output string. Multi-pass approach ensures information is progressively synthesized rather than lost across multiple LLM calls. Termination guaranteed when all data fits within one batch (short-circuits at top with `len(batches) == 1`). Recursion terminates because:\n1. Single-batch case short-circuits at top (`len(batches) == 1`)\n2. Recursive step processes each batch through itself again with same `maxChars`\n\n**Graceful Truncation Policy:** When content exceeds the context window, single oversized items get truncated with `\\n...[truncated]` marker appended; when multiple items exceed limits collectively, each gets capped at `maxChars / len(items)` characters before being processed as one batch. Infinite-loop guard: if batch size equals item count for all items (can't merge), falls back to per-item truncation using `allowedPerItem = maxChars / len(items)`. Still silent β€” no error returned.\n\n##### Overlapping Chunk Support (`chunkTextWithOverlap`)\n\nUtility for splitting raw text into overlapping chunks to maintain context continuity between adjacent segments β€” useful when feeding source material directly into LLMs without pre-processing through the chunking logic. No I/O. Silent swallow: `maxRunes \u003c= 0` β†’ returns full text as single chunk; `overlapRunes \u003e= maxRunes` β†’ clamps to half without warning; final partial chunk reaching end of runes β†’ breaks loop with no error or truncation marker (unlike `reduceItems`).\n\n#### Hierarchical Module Synthesis (`synthesizeNode`, all unexported)\n\nPerforms recursive synthesis of directory structures into hierarchical summaries β€” transforming raw source code and subdirectory metadata into consolidated documentation at each level of a module tree.\n\n**Algorithmic Flow:**\n1. **Context Check β†’ Cache Reuse Gate**: If current node not in `affectedDirs` AND cached module summary exists, return immediately without processing. Short-circuits unaffected directories.\n2. **Recursive Child Synthesis (Bottom-Up)**: Sort children alphabetically, recursively call `synthesizeNode` on each child directory; results collected into map keyed by child name.\n3. **File Processing Loop**: For each file in current node: hash-first cache lookup (if precalculated hash exists and matches cached SHA256 β†’ reuse cached facts without reading disk); read + hash fallback otherwise (read via `tools.ReadFileSafely`, compute SHA-256, update cache entry if already existed with that hash); extraction pipeline on cache miss (chunk content using dynamic limit derived from `c.NumCtx Γ— 4", - "cmd": "# Architecture: `cmd` Package β€” CLI Command Wiring \u0026 Lifecycle Management\n\nThe `cmd` package implements the top-level cobra command hierarchy for Code-Reducer, a documentation-generation agent that writes and maintains project wikis by calling an LLM to produce markdown docs from source code. This module is responsible for registration of subcommands (`init`, `setup`, `update`), resolution of persistent configuration flags, environment validation (Git repo check), signal handling, and delegation to the engine's runner via a shared `executeCommand` function referenced but not defined within this package boundary.\n\n## Data Flow\n\n```\nUser CLI invocation\n β†’ Cobra parses subcommand\n β†’ cmd/init.go : executeCommand(\"init\") β†’ repository scan + wiki scaffolding\n β†’ cmd/setup.go : RunSetupFlow(repoRoot) β†’ interactive config prompts β†’ disk write\n β†’ cmd/update.go : executeCommand(\"update\") β†’ incremental wiki regeneration\n```\n\nAll three subcommand handlers route execution through a single function reference (`executeCommand`) whose implementation is not present in this package. No business logic, validation, or data processing occurs within the registration files themselves; they serve only as wiring between cobra's command framework and the out-of-scope executor.\n\n---\n\n## Root Command Initialization \u0026 Persistent Flags\n\n### `RootCmd` (exported)\n\n- Initialized with `Use=\"code-reducer\"`, short/long descriptions, and `CompletionOptions{DisableDefaultCmd: true}`.\n- Two persistent flags registered via `PersistentFlags`:\n - `--model-id` β€” bound to package-level variable `modelIdFlag`\n - `--num-ctx` β€” bound to package-level variable `numCtxFlag`\n\n### Initialization Sequence in `root.go`\n\n1. **Environment validation** β€” `os.Getwd()` resolves the current working directory; failure is wrapped with `%w` and returned immediately: `fmt.Errorf(\"failed to get current working directory: %w\", err)`.\n2. **Configuration resolution** β€” merges three sources: CLI flags (`--model-id`, `--num-ctx`), environment defaults, and an optional persistent config file (loaded via `internal/config`). The resolved config drives all subsequent behavior; details of this merge are delegated outside the package boundary.\n3. **Implicit setup on first run** β€” if the persistent config file does not exist yet *and* stdin is a terminal (`isatty.IsTerminal(os.Stdin.Fd())` / `isatty.IsCygwinTerminal(os.Stdin.Fd())`), an automatic initial configuration step triggers without requiring user intervention. On non-TTY (CI / piping), this step is skipped and the caller must configure manually via `setup`.\n4. **Mode-gated lifecycle checks** β€” enforced only within the runner/engine layer; at registration time, no validation of project state occurs here. The two modes (`init`, `update`) are mutually exclusive on an already-initialized project; enforcement lives in the external executor.\n5. **Graceful shutdown** β€” installs SIGINT / SIGTERM handlers via `signal.NotifyContext`. A deferred `stop()` cancels the context; any in-flight work that respects the cancelled context will exit. No error-return path exists for signal handling within this file.\n6. **Execution loop** β€” delegates to `engine.NewRunner.Run()` with the resolved config and current directory. The runner emits typed events (`status`, `error`, default).\n\n### Side Effects \u0026 Error Handling in `root.go`\n\n| Operation | Direction | Mechanism | Notes |\n|---|---|---|---|\n| `os.Getwd()` | Read OS state | Filesystem call | Wrapped with `%w`; caller handles failure. |\n| `isatty.IsTerminal(os.Stdin.Fd())` / `isatty.IsCygwinTerminal(os.Stdin.Fd())` | Read OS state | Stdio file descriptor inspection via third-party library (`go-isatty`) | Used only for the guard check before setup flow; result is not persisted. |\n| `fmt.Println(ev.Message)` (2 occurrences) | Write to stdout | Standard library `fmt` package | Triggered inside the callback passed to `runner.Run()` when event type is `\"status\"` or default case. |\n| `fmt.Fprintf(os.Stderr, \"Error: %s\\n\", ev.Message)` | Write to stderr | Standard library `fmt` package | Triggered inside the callback when event type is `\"error\"`. |\n\n**Cobra output suppression:** `RootCmd` has `SilenceErrors: true` and `SilenceUsage: true`. Errors that would otherwise be printed by cobra's default handler are suppressed β€” callers must handle errors explicitly (which this function does). No panic calls observed; all failure paths return errors propagated to cobra.\n\n### Mutable State \u0026 Concurrency in `root.go`\n\n| Variable | Scope | Mutability | Notes |\n|---|---|---|---|\n| `modelIdFlag` | Package-level (`var`) | Mutated once via `StringVar` in `init()` | Set by flag binding; value consumed in `executeCommand()` without further writes. |\n| `numCtxFlag` | Package-level (`var`) | Mutated once via `StringVar` in `init()` | Same pattern as above. |\n\nNo other fields or structs are mutated within this file after their initial assignment. No mutexes, channels, atomic operations, or other concurrency primitives appear here; the signal notification context (`signal.NotifyContext`) is used for cancellation but does not protect shared mutable stateβ€”each `executeCommand` invocation runs sequentially and consumes its own local variables.\n\n---\n\n## Interactive Configuration Setup Flow (`setup.go`)\n\n### Exported Functions\n\n| Function | Signature | Notes |\n|---|---|---|\n| `RunSetupFlow` | `(repoRoot string) error` | Interactive setup flow that generates a `.code-reducer.yaml` configuration file. |\n| `promptStringList` | `(reader *bufio.Reader, promptMsg string, existingList []string) []string` | Reads comma-separated list input from stdin. Returns existing values on I/O error or empty/\"clear\"/\"none\" input. |\n\n### Internal (Unexported) Functions\n\n| Function | Signature | Notes |\n|---|---|---|\n| `init` | `()` | Registers `setupCmd` with `RootCmd`. |\n| `promptString` | `(reader *bufio.Reader, promptMsg string, existingVal string) string` | Reads single-line text input from stdin. Returns existing value on I/O error or empty input. |\n\n### Internal State\n\n| Identifier | Type | Notes |\n|---|------|---|\n| `setupCmd` | `\\*cobra.Command` | The `setup` sub-command registered with the root cobra command. |\n\n### High-Level Algorithmic Flow in `RunSetupFlow`\n\n1. **Resolve existing state** β€” attempt to load any previously saved config from the current working directory via `config.LoadConfig(repoRoot)`. If none exists, all prompts default to built-in constants (Ollama defaults for model ID, base URL, context size; `\"wiki\"` for docs directory).\n2. **Prompt user iteratively** for each setting:\n - LLM Model ID β†’ string\n - Ollama Base URL β†’ string\n - Ollama Context Size β†’ integer (parsed from the current value formatted as a string via `strconv.Atoi(ctxInputStr)`)\n - Custom directories/files to ignore β†’ comma-separated list\n - File extensions to ignore β†’ comma-separated list\n - Documentation directory β†’ string\n3. **Preserve existing values on empty input** β€” if the user presses Enter without typing anything, the previously loaded (or default) value is kept.\n4. **Support reset semantics** β€” for list-type inputs, the literal strings `\"clear\"` or `\"none\"` wipe the list entirely; otherwise comma-splitting produces the new list.\n5. **Error swallowing on read failures** β€” if stdin reading fails during any prompt (`bufio.NewReader(os.Stdin)` + `reader.ReadString('\\n')`), the error is logged to stderr and execution continues using existing values (no rollback, no restart).\n6. **Persist final config** β€” after all prompts complete, a single `config.Config` struct is constructed from user input + preserved state and written to disk via `config.SaveConfig`. The success message references a `.yaml` file path defined in the config package.\n\n### Key Business Rules\n\n- **Non-destructive defaults:** Existing configuration is always preferred over built-in constants; new values only replace what the user explicitly provided (or left as default).\n- **Idempotent re-run:** Running `setup` again when a config already exists produces an incremental update rather than a full overwrite.\n- **Graceful degradation on I/O errors:** Any stdin read failure does not abort setup; it silently falls back to existing state.\n\n### Side Effects \u0026 Error Handling in `setup.go`\n\n**Disk reads (filesystem):**\n- `os.Getwd()` β€” reads current working directory from filesystem. Returns error if unavailable; error is wrapped and propagated: `fmt.Errorf(\"failed to get current working directory: %w\", err)`.\n- `config.LoadConfig(repoRoot)` β€” reads a YAML config file from disk at `repoRoot`. If this fails, `existingCfg` remains `nil` (no panic).\n\n**Disk writes (filesystem):**\n- `config.SaveConfig(repoRoot, newCfg)` β€” writes the new configuration to disk. On error, wraps and returns: `\"error saving configuration: %w\"`. No retry or fallback.\n\n**Stdin reads (user input):**\n- `bufio.NewReader(os.Stdin)` + `reader.ReadString('\\n')` β€” called 6 times total across `promptString` and `promptStringList`: model ID prompt, base URL prompt, context size prompt, custom directories/files to ignore, custom file extensions to ignore, documentation directory.\n\n**Stdout writes:**\n- Welcome banner (`fmt.Println`)\n- Each prompt label (e.g., `\"Enter LLM Model ID [existing]:\"`)\n- Success confirmation: `Configuration successfully saved to local \u003cConfigFileName\u003e file.`\n\n**Stderr writes:**\n- On every `ReadString` failure, prints warning to stderr with the error value and falls back to existing/default values. No log level; just a single-line notice per call site.\n\n### Error Handling Patterns in `setup.go`\n\n| Failure point | Behavior |\n|---|---|\n| `os.Getwd()` | Wrapped into `fmt.Errorf(\"failed to get current working directory: %w\", err)` and returned up the stack. |\n| `config.LoadConfig(repoRoot)` | Nil-check only; if error occurs, existing values default to `OllamaDefault*` constants. No wrapping, no logging, no retry. Effectively swallowed. |\n| `strconv.Atoi(ctxInputStr)` | Nil-check only; on parse failure or non-positive result, falls back to `existingNumCtx`. Swallowed silently. |\n| `reader.ReadString('\\n')` in prompt functions | Written to stderr as `\"Warning: error reading input (%v), \u003cfallback\u003e\"`. Returns existing value (or empty for list). No propagation β€” caller never sees the error. |\n| `config.SaveConfig(repoRoot, newCfg)` | Wrapped with `%w`. Returned up the stack via `RunE` on the cobra command, so cobra will print it to stderr and exit non-zero. This is the only path that surfaces errors to the user without extra wrapping. |\n\n### Notable Observations in `setup.go`\n\n- **No panic anywhere** β€” all I/O failures are handled (silently or with a warning).\n- **Two error strategies:** some paths *propagate* (`Getwd`, `SaveConfig`), others *swallow and fallback* (`LoadConfig`, `Atoi`, prompt readers). This inconsistency is intentional: setup tolerates missing config, but treats save failures as fatal.\n- **No retry logic** β€” `SaveConfig` is called once; if it fails, the entire flow aborts.\n- **Stdin errors are not logged to a structured logger** β€” they go directly to stderr with a single `fmt.Fprintf`.\n\n---\n\n## Update Subcommand Registration (`update.go`)\n\n### Public Surface Area\n\nThis file exports **no public functions**, **no structs**, and **no interfaces**.\n\n### Internal Symbols\n\n| Symbol | Kind | Notes |\n|---|------|---|\n| `updateCmd` | `\\*cobra.Command` | Local variable, not exported |\n| `init()` | function | Lowercase; registered via Go's `init()` convention |\n\n### Business Domain Concept\n\n**Business Rule (stated):** When a project documentation system is updated incrementallyβ€”by scanning changed files since the last documented commitβ€”the wiki pages should be regenerated accordingly.\n\n**High-Level Algorithmic Flow:**\n1. The user invokes an `update` subcommand on the root CLI (`RootCmd`).\n2. Cobra parses and validates the command invocation.\n3. The handler delegates to `executeCommand(\"update\")`, which resolves the actual update logic (implementation not present in this file).\n\n**Notes:** This file only registers the CLI entry point; no business logic, validation, or data processing is contained here.\n\n### Side Effects \u0026 Error Handling in `update.go`\n\n| Concern | Status |\n|---|---|\n| I/O (network/disk/DB) | Not present; delegated to external function (`executeCommand`) whose implementation is not present in this package boundary. |\n| Error wrapping | None β€” raw pass-through of `executeCommand(\"update\")` return value via `RunE`. |\n| Sentinel errors | None created here. |\n| Panic handling | Absent β€” if `executeCommand` panics, it propagates up unhandled (standard cobra behavior). |\n\n---\n\n## Init Subcommand Registration (`init.go`)\n\n### Public Surface Area of `cmd/init.go`\n\n**Exported items:** None.\n\n### Business Domain Concept\n\nCLI-driven documentation scaffolding tool (wiki generator). When the user runs `init`, the system scans the repository and generates an initial set of wiki markdown pages. This is a one-time setup operation for project documentation.\n\n### Algorithmic Flow\n\n1. User invokes the `init` subcommand on the CLI.\n2. The command delegates to `executeCommand(\"init\")`.\n3. That function (defined elsewhere in this module) performs the repository scan and wiki page generation.\n\n**Note:** All substantive logic is deferred to `executeCommand`, which is not included in this file. This file serves only as registration/wiring for the subcommand within the cobra command hierarchy.\n\n### Side Effects \u0026 Error Handling in `init.go`\n\n| Pattern | Used? |\n|---|---|\n| Wrap / aggregate errors | ❌ |\n| Sentinel error | ❌ |\n| Panic | ❌ |\n| Pass-through delegation | βœ… β€” raw result of `executeCommand(\"init\")` returned directly with no transformation. |\n\n### Mutable State \u0026 Concurrency in `init.go`\n\n**No mutable state.** The only variable (`initCmd`) is initialized once at package scope and never reassigned. No concurrent access or modification occurs within this file.\n\n---\n\n## Package-Level Summary\n\n| Concern | Implementation Strategy |\n|---|---|\n| Command registration | Each subcommand file declares a local `\\*cobra.Command` variable, registers it via `init()` (package-level), and wires its `RunE` callback to delegate to the shared `executeCommand(string) error` reference. |\n| Configuration persistence | Handled by an external config package (`internal/config`). The `cmd` package only triggers the interactive flow in `setup.go` or reads flags in `root.go`. |\n| Engine execution | Delegated to `engine.NewRunner.Run()`; event callbacks are defined inline in `root.go` for stdout/stderr routing. |\n| Concurrency model | Sequential-only within this package boundary. Signal handling uses a cancellation context but does not protect shared mutable state. No channels, mutexes, or atomic types appear anywhere in the four files. |\n| Error propagation strategy | Mixed: some paths propagate raw errors (`init`, `update`), others wrap and return sentinel messages (`root.go`'s git check), and still others swallow I/O failures silently with fallback values (`setup.go`). The only path that reaches cobra's default error printer is the final save failure in `setup.go`. |", - "internal": "# Architecture β€” Internal Packages\n\n## Module Responsibility \u0026 Data Flow\n\nThe `internal` directory implements four orthogonal subsystems for the code analysis tool: **configuration resolution** (`config`), **pipeline orchestration** (`engine`), **sandbox enforcement** (`security`), and **filesystem/Git operations** (`tools`). Each subsystem has a single responsibility; cross-subsystem communication occurs only through the `*config.Config` value passed to engine entry points, which is read-only within `internal/engine`.\n\n---\n\n## Package: config β€” Multi-Layer Configuration Resolution\n\n### Responsibility\n\nThe `config` package implements a four-tier precedence pipeline for resolving all configurable values. Each layer overwrites prior values when present; missing layers are transparently skipped. Three properties are resolved independently (`ModelID`, `OllamaBaseURL`, `OllamaNumCtx`). The package also structures the analysis pipeline into named **extraction steps** (`ExtractionStep`), each carrying a distinct prompt template.\n\nAll non-structural state (resolved config, local variables) is ephemeralβ€”scoped to function calls and returned as values. No package-level mutable state survives beyond initialization. Concurrency primitives are absent; the package is single-threaded in design.\n\n### Data Flow\n\n```\nDefaultExtractionSteps (init), DefaultIgnores, DefaultIgnoredExtensions\n β”‚\n β–Ό\nLoadConfig(cwd) ──► *Config{ModelID, OllamaBaseURL, OllamaNumCtx, ExtractionSteps, ...}\n β”‚ β”‚\n β”‚ β”œβ”€β”€ defaults (hard-coded constants)\n β”‚ β”œβ”€β”€ os.Getenv for CODE_REDUCER_MODEL_ID, OLLAMA_BASE_URL, OLLAMA_NUM_CTX\n β”‚ └── strconv.Atoi on modelIdFlag, numCtxFlag (err==nil \u0026\u0026 n\u003e0 required)\n β”‚\n β–Ό\n*Config returned to caller; no package-level mutation persists\n```\n\n### Types\n\n#### `ExtractionStep`\n\n```go\ntype ExtractionStep struct {\n Name string // step identifier (e.g., \"api-signatures\", \"business-logic\")\n Prompt string // prompt template applied to this phase during analysis\n}\n```\n\nInstances are held in a package-level slice (`DefaultExtractionSteps`). The slice is immutable after declaration; callers receive copies or slices of it.\n\n#### `Config`\n\n```go\ntype Config struct {\n ModelID string // resolved model identifier (last-wins: default β†’ YAML β†’ env β†’ flag)\n OllamaBaseURL string // resolved base URL (default β†’ YAML β†’ env; CLI flag omitted)\n OllamaNumCtx int // resolved context size, validated positive (default β†’ YAML β†’ env β†’ flag)\n DocsDir string // path to the documents directory under analysis\n ExtractionSteps []ExtractionStep // ordered list of extraction phases\n Ignore []string // absolute paths excluded from traversal\n IgnoreExtensions []string // file extensions excluded from traversal\n}\n```\n\n### Constants and Defaults\n\n#### Environment Variable Keys\n\n| Constant | Value | Purpose |\n|---|---|---|\n| `CodeReducerModelIdEnvKey` | `\"CODE_REDUCER_MODEL_ID\"` | Overrides `Config.ModelID` when set. |\n| `OllamaBaseUrlEnvKey` | `\"OLLAMA_BASE_URL\"` | Overrides `Config.OllamaBaseURL` when set. |\n| `OllamaNumCtxEnvKey` | `\"OLLAMA_NUM_CTX\"` | Overrides `Config.OllamaNumCtx`; value must parse as a positive integer. |\n\n#### Default Values\n\n| Constant | Value | Purpose |\n|---|---|---|\n| `OllamaDefaultBaseURL` | `\"http://localhost:11434\"` | Fallback base URL when no prior layer supplies one. |\n| `OllamaDefaultModelID` | `\"ornith:9b\"` | Default model identifier. |\n| `OllamaDefaultNumCtx` | `8192` | Default context size. |\n\n#### Configuration File\n\n| Constant | Value | Purpose |\n|---|---|---|\n| `ConfigFileName` | `.code-reducer.yaml` | Filename resolved relative to the current working directory by `getConfigPath`. |\n\n### Package-Level Immutable Defaults\n\nThree slices are declared at package init and never mutated:\n\n```go\nvar DefaultIgnores []string // system-supplied ignore paths\nvar DefaultIgnoredExtensions []string // system-supplied ignored extensions\nvar DefaultExtractionSteps []ExtractionStep // ordered extraction phases\n```\n\nCallers may append to these slices externally, but the package itself never reassigns them. `SaveConfig` writes the current slice state back to disk; callers must mutate before save if they need custom defaults.\n\n### Public API Surface\n\n#### Path Resolution and Existence Check\n\n##### `getConfigPath(cwd string) (string, error)`\n\nConstructs the full path to `.code-reducer.yaml` by joining `cwd` with `ConfigFileName`. Returns a non-nil error only on I/O failure during construction; normal cases return `nil`.\n\n##### `ConfigExists(cwd string) bool`\n\nReturns whether the configuration file exists at the resolved path. Internally calls `os.Stat`; all errors (permission denied, I/O failures, etc.) are swallowed and treated as \"does not exist\". The function returns no error.\n\n#### Configuration Loading\n\n##### `LoadConfig(cwd string) (*Config, error)`\n\nReads and parses the YAML configuration file at the resolved path. Error handling:\n- File-read failure (`os.ReadFile`) β†’ returned to caller unchanged (pass-through).\n- YAML parse failure β†’ wrapped as `\"failed to parse yaml config: %w\"`.\n\nReturns a non-nil `*Config` on success; the struct is initialized with zero values before parsing, so callers receive a populated object even when the file contains only valid empty mappings.\n\n##### `ResolveConfig(repoRoot, modelIdFlag, numCtxFlag string) (*Config, error)`\n\nTop-level entry point for full configuration resolution. Algorithm:\n1. Calls `LoadConfig` to read the YAML file. If the error is a non-existent-file sentinel (`os.IsNotExist(err)`), it swallows and initializes an empty `Config{}`; all other errors propagate wrapped as `\"failed to load configuration file:\"`.\n2. Layers defaults (hard-coded constants) over the loaded config.\n3. Layers environment variable overrides using `os.Getenv` for `CODE_REDUCER_MODEL_ID`, `OLLAMA_BASE_URL`, and `OLLAMA_NUM_CTX`. Non-empty values are used; empty strings fall through. Env read errors are swallowed.\n4. Applies CLI flags (`modelIdFlag`, `numCtxFlag`). The flag value is parsed via `strconv.Atoi`; only `err == nil \u0026\u0026 n \u003e 0` is acceptedβ€”invalid or non-positive values cause the prior layer's value to be retained silently.\n\nReturns a fully resolved `*Config`. Errors are either pass-through (from `LoadConfig`) or wrapped with `\"failed to load configuration file:\"`. Notable: if YAML is malformed, the parse error wraps and propagates; otherwise resolution always succeeds for missing files.\n\n#### Configuration Persistence\n\n##### `SaveConfig(cwd string, cfg *Config) error`\n\nSerializes the config struct back to disk using `os.WriteFile` with `0600` permissions. Error handling:\n- Write success β†’ returns `nil`.\n- Parse failure (marshal YAML) β†’ wrapped as `\"failed to marshal yaml: %w\"`.\n- File-write failure β†’ wrapped as `\"failed to write config file: %w\"`.\n\nWrites directly (no temp-file + rename); a partial write can leave a truncated file on disk. Permission normalization is implicitβ€”existing files with different permissions will fail the write.\n\n---\n\n## Package: engine β€” Code Documentation Pipeline Engine\n\n### Responsibility\n\nThe `internal/engine` package implements a Map-Reduce pipeline that generates and maintains technical documentation for a Go codebase using Ollama-compatible LLM inference endpoints. The engine produces three artifacts per run: `architecture.md` (global overview), `quickstart.md` (onboarding guide), and per-module API summaries under `modules/`. Execution is gated by a repository-level lock to prevent concurrent runs against the same root, and state persistence across invocations is handled by an on-disk metadata cache.\n\n### Data Flow\n\n```\nsecurity.EnsureGitignoreHasLockfile(repoRoot) ──► security.AcquireLock(repoRoot) ──► defer Unlock()\n β”‚ β”‚\n β–Ό β–Ό\n*LLMClient from config.ModelID, BaseURL, NumCtx RunInit(ctx, repoRoot, cfg, onEvent)\n β”‚ β”‚\n β–Ό β–Ό\nclient.RunInit / client.RunUpdate β†’ synthesizeNode (bottom-up tree reduction) β†’ CallLLM per step/chunk\n β”‚\n β–Ό\ntools.WriteFileSafely for architecture.md, quickstart.md, modules/*.md\n```\n\n### Pipeline Orchestration\n\n#### Runner Entry Point (`runner.go` β€” `Runner.Run`)\n\nThe public entry point for all pipeline invocations. Responsibilities: repository isolation, concurrency guard acquisition, LLM client construction from config, and mode-gated dispatch to the appropriate pipeline method.\n\n```\nRun(ctx, repoRoot, mode string, onEvent func(Event)) error\n β”œβ”€ ensure .gitignore lockfile entry (best-effort; failure logged as warning)\n β”œβ”€ acquire repository-level lock via security.AcquireLock(repoRoot)\n β”‚ └─ defer lock.Unlock() β€” released on any return path including errors\n β”œβ”€ build *LLMClient from r.cfg.ModelID, BaseURL, NumCtx\n β”œβ”€ switch mode:\n β”‚ case \"init\": client.RunInit(ctx, repoRoot, r.cfg, onEvent)\n β”‚ case \"update\": client.RunUpdate(ctx, repoRoot, r.cfg, onEvent)\n β”‚ default: return fmt.Errorf(\"unsupported mode: %s\", mode)\n```\n\nThe `cfg *config.Config` field is read-only within this file; all mutations to the runner's configuration originate before construction. The lock returned by `security.AcquireLock(repoRoot)` is a local variable with deferred cleanupβ€”its internal type and mutual exclusion semantics are not observable from this source alone.\n\n#### Orchestrator Pipelines (`orchestrator.go` β€” `RunInit`, `RunUpdate`)\n\n##### Full Run (`RunInit`)\n\nExecutes an all-or-nothing regeneration regardless of cache state:\n\n1. **Setup** β€” Load `.gitignore` via `tools.LoadGitignore(repoRoot)`; merge with user ignore list; load metadata cache from disk via `cache.loadMetadataCache()`. Failures logged as warnings, not returned.\n2. **Map Phase** β€” Discover code files via `tools.DiscoverCodeFiles(...)`, compute SHA-256 per file via `computeSHA256(repoRoot, f)`, build directory tree via `buildTree(codeFiles)`. Hash computation errors silently skipped in loops; only stored if hash computed successfully.\n3. **Mark All Affected** β€” Recursively flag every node as affected (init starts fresh).\n4. **Reduce Phase** β€” Call `synthesizeNode` bottom-up on the tree to produce a root summary string and per-module summaries via LLM calls through `c.CallLLM`.\n5. **Generate Standard Docs** β€” Produce `architecture.md` and `quickstart.md` from the root summary via two additional `CallLLM` invocations: one for architecture, one for quickstart. Writes via `tools.WriteFileSafely`; errors wrapped with context (e.g., `\"failed to write quickstart.md: %w\"`).\n6. **Update AGENTS.md** β€” Read existing file via `tools.ReadFileSafely(repoRoot, \"AGENTS.md\")`. If missing, create with guidelines header. If present but does not contain \"AI Agent Guidelines\", append separator + new content block. Only writes when content actually differs. Errors: if read fails but write succeeds β†’ `\"failed to write AGENTS.md: %w\"`; if both fail β†’ wrapped errors chain.\n7. **Teardown** β€” Persist metadata cache via `saveMetadataCache(repoRoot, docsDir, cache)`. Failure logged as warning, function always returns nil at end regardless of this failure.\n\n##### Incremental Run (`RunUpdate`)\n\nExecutes only re-generations for directories whose contents actually changed:\n\n1. **Setup** β€” Load pipeline state from metadata cache (same path as init).\n2. **Map Phase** β€” Discover code files, hash each via `computeSHA256`, build tree.\n3. **Change Detection** β€” Compare current file hashes against cached hashes in `cache.Files`:\n - New file in codebase but absent from cache β†’ `FileChange{Status: \"Added\"}`\n - Existing file with different hash β†’ `FileChange{Status: \"Modified\"}`\n - Cached file no longer in codebase β†’ `FileChange{Status: \"Deleted\"}` (also removed from cache)\n4. **Prune Stale Modules** β€” Remove any module summaries whose directory is no longer active, delete their markdown files on disk via `os.Remove(absModuleFile)` (error assigned to `_`, never logged).\n5. **Propagate Affected** β€” Run tree-aware propagation (`propagateAffected`/`determineAffected`) to mark directories affected by changes. Errors from these calls are discarded without capture.\n6. **Early Exit** β€” If zero directories are affected β†’ pipeline ends with \"up to date\" status, no further synthesis occurs.\n7. **Conditional Reduce** β€” If root-level change detected or global docs don't exist on disk (`os.Stat(absArch)`/`os.Stat(absQs)`), run full synthesis; otherwise skip and reuse existing `architecture.md`/`quickstart.md`.\n8. **Teardown** β€” Save updated cache via `saveMetadataCache(repoRoot, docsDir, cache)`.\n\n### Directory Tree Construction \u0026 Affected Propagation (`tree.go`)\n\n#### Type Definitions\n\n| Type | Fields | Purpose |\n|------|--------|---------|\n| `FileChange` | `Path string`, `Status string` | Models a file operation with path and status (`Added`, `Modified`, `Deleted`). Consumed by `determineAffected`. |\n| `DirNode` | `Path string`, `Files []string`, `Children map[string]*DirNode` | Hierarchical tree node holding files at that level and child directories in its subtree. Produced by `buildTree`. |\n\n#### Functions (all unexported)\n\n- **`buildTree(codeFiles)`** β€” Converts a flat list of file paths into nested `*DirNode` structure, preserving directory nesting by splitting on `/`. Appends to `DirNode.Files`; populates `DirNode.Children` via map assignment.\n- **`propagateAffected(tree, changeset)`** β€” Traverses the tree recursively: if any direct child of the current node is in the changeset, mark it affected. Accumulates affected status upward through ancestors. Returns mutated `affectedDirs`.\n- **`determineAffected(tree, changeset)`** β€” Computes module path for each directory via `ToSafeMarkdownFilename`, checks existence on disk via `os.Stat(absModulePath)`, reads metadata cache (`cache.Modules[n.Path] == \"\"`), marks node affected if any check fails. Only `os.IsNotExist` is acted on; all other errors are silently dropped.\n\n---\n\n## Package: engine β€” LLM Client Abstraction \u0026 Response Parsing\n\n### Type: `LLMClient`\n\n```go\ntype LLMMClient struct {\n ModelID string\n BaseURL string\n NumCtx int\n HTTPClient *http.Client // default transport, no custom config (no proxy/redirect/TLS)\n}\n```\n\nConstructed via `NewLLMClient(modelID, baseURL, numCtx)` with a 10-minute HTTP timeout. Fields set once; never reassigned within any method.\n\n### Methods on `*LLMClient`\n\n#### `CallLLM(ctx context.Context, systemPrompt string, messages []Message, jsonFormat bool) (string, error)`\n\nFull lifecycle: prepends system prompt as first message with role `\"system\"` β†’ serializes into Ollama `/api/chat` JSON payload with model ID and context window size β†’ non-streaming HTTP POST β†’ response body fully read on 200 OK or truncated at 1 KB via `io.LimitReader(resp.Body, 1024)` on non-OK status. Returns only assistant content string; returns empty string on failure.\n\n**Error behavior:**\n\n| Failure path | Error type | Wrapping strategy |\n|---|---|---|\n| JSON marshal fails in request prep | `fmt.Errorf(\"failed to marshal request: %w\", err)` | Wrapped with `%w` |\n| `http.NewRequestWithContext` fails | raw error from `NewRequestWithContext` | **Not wrapped** β€” passed through as-is |\n| HTTP Do (network-level failure) | raw error from `c.HTTPClient.Do(req)` | **Not wrapped** |\n| Body read fails on 200 OK path | raw error from `io.ReadAll` | **Not wrapped** |\n| JSON unmarshal fails on response | `fmt.Errorf(\"failed to parse response: %w\", err)` | Wrapped with `%w` |\n| Non-OK status (body truncated at 1 KB) | `fmt.Errorf(\"ollama api error: status %d, response: %s\", ...)` | New error constructed; body discarded if read fails |\n\n#### `GetBaseSystemPrompt() string`\n\nReturns the fixed base role definition and defensive guidelines inherited by every LLM call regardless of task type.\n\n#### `GetDefaultSystemPrompt(command string) string`\n\nAppends command-specific instructions based on the command string: two named variants exist (`module_synthesis`, `architecture`); any other value receives only the base layer.\n\n### Response Parsing Layer (`json_parser.go`)\n\nInternal helper `extractBalancedJSON` is lowercase and excluded from public surface area.\n\n#### Business Rules\n\n1. Markdown code fences (` ``` `) surrounding JSON must be stripped before parsing β€” handled by `StripOuterMarkdownFence`.\n2. Balanced bracket matching required β€” strings containing `{`, `[`, `}`, `]` literals (including escaped characters like `\\n`) must not break extraction logic β€” handled by internal rune-level scanner with escape state tracking inside string literals.\n3. When multiple balanced JSON candidates exist in a single response, each must be attempted individually until one succeeds, or the entire trimmed string is tried as a final fallback β€” handled by `CleanJSONResponse`/`UnmarshalJSONResponse`.\n\n#### Functions\n\n| Function | Behavior | Error surface | Swallowed? |\n|---|---|---|---|\n| `StripOuterMarkdownFence(s string) string` | Strips surrounding markdown fences; falls back to trimmed input if none match | None β€” returns string only | N/A (no error return) |\n| `CleanJSONResponse(s string) string` | Iterates stripped content, calls `extractBalancedJSON`, returns first successful candidate or original trimmed input unchanged | None β€” returns string only | Yes β€” unbalanced JSON inside response is swallowed silently |\n| `UnmarshalJSONResponse(s string, target interface{}) error` | Multi-layered attempt: (1) collect balanced candidates, try each into target; keep last error in `lastErr`; (2) if any succeeded, return nil; otherwise try entire trimmed string; (3) terminal errors β€” `\"failed to unmarshal JSON candidates: %w\"` or `\"no valid JSON found in response\"` | Explicit (`error`) | Partial parse failures collected and re-emitted via `%w` on first path; \"no valid JSON\" covers second path |\n\n---\n\n## Package: engine β€” Synthesis Engine \u0026 Chunking\n\n### Context-Aware Chunking (`chunking.go` β€” all unexported)\n\nThe system processes extracted code/fact items in batches constrained by LLM context limits (`NumCtx * 3`). Three distinct processing modes exist:\n\n#### Module Synthesis (`reduceInChunks`)\n\nSynthesizes architectural-level nodes into architecture descriptions using a `\"module_synthesis\"` system prompt. Each batch represents evidence contributing to understanding the module's design intent. Response passes through `StripOuterMarkdownFence`. Errors wrapped with `\"LLM error during synthesis: %w\"`.\n\n#### File Fact Consolidation (`reduceFileFacts`)\n\nConsolidates extracted facts about a specific step within a file with deduplication and merging. System prompt built locally by concatenating `c.GetBaseSystemPrompt()` + fixed role string. Response also passes through `StripOuterMarkdownFence`. Errors wrapped with `\"LLM error during file fact consolidation: %w\"`.\n\n#### Recursive Convergence (`reduceItems`)\n\nCore business rule: reduce items in batches β†’ collect intermediate results β†’ recursively apply reduction until everything converges into a single output string. Multi-pass approach ensures information is progressively synthesized rather than lost across multiple LLM calls. Termination guaranteed when all data fits within one batch (short-circuits at top with `len(batches) == 1`). Recursion terminates because:\n1. Single-batch case short-circuits at top (`len(batches) == 1`)\n2. Recursive step processes each batch through itself again with same `maxChars`\n\n**Graceful Truncation Policy:** When content exceeds the context window, single oversized items get truncated with `\\n...[truncated]` marker appended; when multiple items exceed limits collectively, each gets capped at `maxChars / len(items)` characters before being processed as one batch. Infinite-loop guard: if batch size equals item count for all items (can't merge), falls back to per-item truncation using `allowedPerItem = maxChars / len(items)`. Still silent β€” no error returned.\n\n#### Overlapping Chunk Support (`chunkTextWithOverlap`)\n\nUtility for splitting raw text into overlapping chunks to maintain context continuity between adjacent segmentsβ€”useful when feeding source material directly into LLMs without pre-processing through the chunking logic. No I/O. Silent swallow: `maxRunes \u003c= 0` β†’ returns full text as single chunk; `overlapRunes \u003e= maxRunes` β†’ clamps to half without warning; final partial chunk reaching end of runes β†’ breaks loop with no error or truncation marker (unlike `reduceItems`).\n\n### Hierarchical Module Synthesis (`synthesize.go` β€” `synthesizeNode`, all unexported)\n\nPerforms recursive synthesis of directory structures into hierarchical summariesβ€”transforming raw source code and subdirectory metadata into consolidated documentation at each level of a module tree.\n\n**Algorithmic Flow:**\n1. **Context Check β†’ Cache Reuse Gate**: If current node not in `affectedDirs` AND cached module summary exists, return immediately without processing. Short-circuits unaffected directories.\n2. **Recursive Child Synthesis (Bottom-Up)**: Sort children alphabetically, recursively call `synthesizeNode` on each child directory; results collected into map keyed by child name.\n3. **File Processing Loop**: For each file in current node: hash-first cache lookup (if precalculated hash exists and matches cached SHA256 β†’ reuse cached facts without reading disk); read + hash fallback otherwise (read via `tools.ReadFileSafely`, compute SHA256, update cache entry if already existed with that hash); extraction pipeline on cache miss (chunk content using dynamic limit derived from `c.NumCtx Γ— 4 characters Γ— 0.75` with minimum of 512 tokens and max overlap cap; for each chunk iterate over every step in `cfg.ExtractionSteps`, sending to LLM with system prompt built from base + step-specific prompt; consolidate all chunks for that step via `reduceFileFacts`; append consolidated result under step's name heading).\n4. **Component Assembly**: After processing files, append any non-empty child summaries to components list. If no components exist at all (no files AND no children), clear module cache and return empty string.\n5. **Final Synthesis \u0026 Persistence**: Pass assembled components to `reduceInChunks` for directory path; store result in both memory cache (`cache.Modules[node.Path]`) and on disk at `\u003crepoRoot\u003e/\u003cdocsDir\u003e/modules/\u003csafe-filename\u003e` via `tools.WriteFileSafely`; return final summary string.\n\n**Domain concepts:** Hierarchical module synthesis (bottom-up: files contribute facts, subdirectories contribute their own summaries, parent combines both); context-aware truncation (file content limits scale with `c.NumCtx`, reserving 75% of tokens for file data and 25% for prompts/output); two-level caching (files cache individual facts keyed by SHA256, modules cache full synthesized summaries keyed by path; reuse only when node not affected); multi-step extraction pipeline (each file passes through configurable sequence of LLM steps, each producing independent fact set consolidated before combining with other files' results).\n\n**Error handling in `synthesizeNode`:**\n\n| Source | Pattern | Propagation |\n|---|---|---|\n| Context cancel | Direct return at top and inside per-file loop (`if err := ctx.Err(); err != nil { return \"\", err }`) | Upward, no wrapping |\n| LLM call failure | Wrapped: `\"LLM error extracting %s for %s: %w\"` with step name + file path | Upward |\n| File read failure | Logged via `logEvent(\"status\", ...)`, skipped silently; no hash/facts/anything added to components | None (swallowed) |\n| Module write failure | Wrapped: `\"failed to write module documentation for %s: %w\"` with dir path | Upward |\n| Recursive child error | Propagated as-is (`return \"\", err`) | Upward |\n| Empty components | Returns `(\"\", nil)`, clears `cache.Modules[node.Path] = \"\"` | None (clean return) |\n\n---\n\n## Package: engine β€” Filesystem State Management \u0026 Utilities\n\n### Type: `MetadataCache`\n\n```go\ntype MetadataCache struct {\n Files map[string]FileCacheEntry // virtual path β†’ cached SHA256 + facts\n Modules map[string]string // module identifier β†’ summary string\n}\n```\n\nBoth maps initialized with `make()` when nil; populated during load/save lifecycle; serialized to disk. No in-memory locking applied within this file.\n\n### Functions (all unexported except `IsInitialized`)\n\n#### Core Concept: Filesystem-Based State Cache\n\nPersists an in-memory cache to disk so expensive computations can be avoided across runs. Stored as `docsDir/.metadata.json`, holds two pieces of state:\n- **File-level metadata** (`Files map`): maps virtual paths (filenames) to cached SHA256 hashes plus associated \"facts\" strings.\n- **Module-level metadata** (`Modules map`): maps module identifiers to string values.\n\n#### Functions\n\n| Function | Direction | Path constructed | Notes |\n|---|---|---|---|\n| `loadMetadataCache()` | Read | `docsDir/.metadata.json` | Attempts to read JSON; if absent, returns freshly initialized empty cache (not error); if present but unparseable, errors. Null maps normalized to empty on load. |\n| `IsInitialized(repoRoot, docsDir string) bool` | Read (existence probe) | `docsDir/.metadata.json` | Probes for existence of `.metadata.json`. Successful read β†’ true; any non-nil error β†’ false. N/A for missing file (returns false). |\n| `saveMetadataCache()` | Write | `docsDir/.metadata.json` | Serializes struct back to JSON with indentation, writes atomically via `tools.WriteFileSafely`. Returns raw errors from marshal/write unchanged. |\n| `computeSHA256(virtualPath)` | Read (virtual path) | caller-supplied virtual path | Reads file by virtual path, runs SHA-256 on content, returns hex-encoded digest. No disk I/O within this file; pure in-memory (`crypto/sha256`). |\n\n**Error handling patterns:**\n\n| Function | On missing cache file | On generic read/write error | On JSON parse error |\n|---|---|---|---|\n| `loadMetadataCache` | Returns empty initialized `*MetadataCache`, nil error | Wraps with `\"failed to read metadata cache: %w\"` | Wraps with `\"failed to unmarshal metadata cache: %w\"` |\n| `IsInitialized` | N/A (returns false) | N/A β€” any non-nil error β†’ false | β€” |\n| `saveMetadataCache` | β€” | Returns raw error from `tools.WriteFileSafely` | Returns raw error from `json.MarshalIndent` |\n| `computeSHA256` | β€” | Returns raw error from `tools.ReadFileSafely` | N/A (no parsing) |\n\n### Function: `ToSafeMarkdownFilename(modulePath string) string`\n\nPure string transformation with no business logic or complex algorithmic flow. Maps a module path to a safe filename for markdown documentation.\n\n**Conversion pipeline:**\n1. Replace every `/` character in input `modulePath` with `_`.\n2. If result is empty or equals `\".\"`, substitute with `\"root\"`.\n3. Append `.md` and return.\n\n**Domain concept:** Module-to-document routing β€” given a module identifier, produce corresponding markdown file path. Assumes caller already owns consistent `modulePath` convention; this utility only adapts that path for filesystem-safe filenames.\n\n---\n\n## Package: security β€” Sandbox Enforcement \u0026 Process Locking\n\n### Responsibility\n\nThe `security` package encapsulates three orthogonal concerns for code-reducer processes: **sandbox enforcement** (preventing path traversal beyond the repository boundary), **process-level mutual exclusion** via file-based locks, and **operational hygiene** (keeping lock state out of version control). No networking, database access, or external process communication occurs within this package.\n\nAll functions are pure-Go with no global mutable state. Every observable side effect is local to a single `*SimpleLock` receiver instance or the calling goroutine's stack. Errors follow a strict wrapping convention: security-critical violations and actionable sentinel messages return unwrapped plain strings; standard-library errors use `%w`; best-effort cleanup paths swallow non-fatal I/O failures silently.\n\n**Data flow** proceeds in two independent tracks that share no state except through the repository root path string:\n\n1. **Path resolution track:** `SafeResolve` β†’ called by `AcquireLock` to canonicalize the lock file location before attempting acquisition.\n2. **Lock lifecycle track:** `AcquireLock` β†’ returns a `*SimpleLock` receiver β†’ caller uses it for work β†’ calls `Unlock()` for release β†’ `EnsureGitignoreHasLockfile` runs during cleanup to append the lock filename to `.gitignore`.\n\n### Types\n\n#### `SimpleLock`\n\nFile-based process lock with internal mutex protection. The struct is constructed by `AcquireLock`; once acquired, `lockPath` and `file` are set exactly once under mutex scope, then only modified during `Unlock()`.\n\n| Field | Type | Semantics |\n|-------|------|-----------|\n| `lockPath` | `string` | Absolute path to the `.code-reducer.lock` file. Set in constructor; zero-valued if acquisition failed or was never called. |\n| `file` | `*os.File` | Opened lock file handle. Acquired with `O_CREATE\\|O_EXCL`; set to `nil` inside `Unlock()`. |\n| `mu` | `sync.Mutex` | Per-instance mutex protecting `closed` and the pointer-clear of `file` during unlock. |\n| `closed` | `bool` | Transition flag: `false` while locked, flips to `true` once `Unlock()` completes its write operations (under mutex). |\n\n### Public API Surface\n\n#### Functions\n\n##### `SafeResolve(repoRoot, inputPath string) (string, error)`\n\nComputes the absolute path of an input relative to a repository root. The resolution is symlink-aware: ancestor components are evaluated through their physical targets via `filepath.EvalSymlinks` before being walked upward until one succeeds. The final result must lie strictly inside the resolved rootβ€”any escape beyond it returns an unwrapped string error indicating a security violation.\n\n##### `AcquireLock(repoRoot string) (*SimpleLock, error)`\n\nResolves the lock file path through `SafeResolve`, then attempts atomic creation with `O_CREATE\\|O_EXCL`. On success, writes the process PID to the newly created file and returns a populated `*SimpleLock`. If the file already exists (anewer process holds it), returns an unwrapped sentinel string describing that another process has acquired the lock.\n\n##### `EnsureGitignoreHasLockfile(repoRoot string) error`\n\nReads `.gitignore`; if present, appends `# Code-Reducer Lockfile\\n.code-reducer.lock\\n` only when not already contained in the file. If `.gitignore` does not exist, creates it with append semantics (`O_APPEND\\|O_CREATE\\|O_WRONLY`). Errors from reading or writing are wrapped; non-existence of `.gitignore` is treated as expected and returned as `nil`.\n\n#### Methods\n\n##### `(l *SimpleLock).Unlock() error`\n\nIdempotent release: acquires the internal mutex, closes the file handle (discarding any close-error), removes the lock file from disk via best-effort `os.Remove`, sets `closed = true`, then releases the mutex. A second call is safe and returns `nil`. Swallowed errors for cleanup operations are discarded; removal failures that occur after a successful close surface as wrapped errors only in the specific narrow case where both close-succeeded-and-remove-failedβ€”but this path effectively swallows non-`IsNotExist` remove errors in practice.\n\n### Business Rules \u0026 Domain Concepts\n\n#### Repository Boundary Enforcement\n\nAll internal paths must resolve strictly within `repoRoot`. Any resolution that escapes beyond it returns a standalone string error: `\"security violation: path traversal detected: %q\"`, never wrapped. This is the one unwrapped class of non-sentinel errors in the packageβ€”security-critical, so callers detect it via exact match or `strings.Contains` rather than type assertion.\n\n#### Symlink-Aware Path Resolution\n\nAncestors are evaluated through their physical targets before path reconstruction. This prevents traversal attacks that use symlinked directories to bypass the sandbox while preserving logical input structure for legitimate uses. Implementation: `filepath.EvalSymlinks(absRoot)` resolves the root once; subsequent ancestor walks use `os.Lstat` with a fallback to `EvalSymlinks` on each component until one exists, then rebuild upward from that point.\n\n#### Process-Level File Locking\n\nThe lock file is `.code-reducer.lock`. Acquisition uses `O_EXCL` for atomic creationβ€”no two processes can hold the lock simultaneously on the same filesystem. On success, the PID of the acquiring process is written as a single-line string (`fmt.Sprintf(\"%d\\n\", os.Getpid())`). The lockfile content therefore serves as both a record and a mechanism for stale detection: if another process exits without releasing its lock, a new acquisition attempt detects the existing file and reports an actionable error instructing manual cleanup.\n\n#### Stale Lock Detection\n\nIntentional behaviorβ€”the system does not attempt to detect or kill the holding process. The caller receives an unwrapped string describing that the lock is held by another process and must clean up manually. This keeps the package non-invasive; it never interacts with external processes.\n\n#### Automated Gitignore Integration\n\nThe lockfile path is appended to `.gitignore` during unlock so it does not enter version control. Idempotent: if a line containing `code-reducer.lock` already exists, no duplicate entry is written. The lock filename itself is the only thing tracked; comment prefix provides human readability without affecting git behavior.\n\n---\n\n## Package: tools β€” Filesystem \u0026 Git Operations\n\n### Responsibility\n\nThis module provides two complementary interfaces for repository-aware operations: safe file I/O with TOCTOU race mitigation, and Git process abstraction for executing commands within the working tree. Both modules operate on a single root path (`repoRoot`) without shared mutable state, and neither uses concurrency primitives β€” all functions are goroutine-safe by construction.\n\n### `file_tools.go` β€” Safe File I/O \u0026 Discovery\n\n#### Core Functions\n\n| Function | Input | Output |\n|---|---|---|\n| [`ReadFileSafely`](file:///path/to/file_tools.go#L15-L64) | `repoRoot`, `virtualPath` `string` | `[]byte`, `error` |\n| [`WriteFileSafely`](file:///path/to/file_tools.go#L70-L132) | `repoRoot`, `virtualPath` `string`; `content` `[]byte` | `error` |\n| [`LoadGitignore`](file:///path/to/file_tools.go#L138-L164) | `repoRoot` `string` | `[]string`, `error` |\n| [`ShouldIgnoreFile`](file:///path/to/file_tools.go#L170-L236) | `repoRoot`, `relPath` `string`; `gitIgnore` `*ignore.GitIgnore`; `ignoredExtensions` `[]string` | `bool` |\n| [`DiscoverCodeFiles`](file:///path/to/file_tools.go#L242-L291) | `repoRoot` `string`; `ignores`, `ignoredExtensions` `[]string` | `[]string`, `error` |\n| [`IsBinaryFile`](file:///path/to/file_tools.go#L297-L315) | `path` `string` | `bool` |\n\n#### TOCTOU Race Mitigation Pattern\n\nBoth `ReadFileSafely` and `WriteFileSafely` implement a Time-Of-Check-Time-Of-Use safety pattern:\n\n1. Resolve the virtual path to an absolute safe path via `SafeResolve`\n2. Open the file descriptor first\n3. Perform `lstat` on the resolved path (no symlink following)\n4. Perform `fstat` on the open file descriptor (follows symlinks)\n5. Verify that `lstat` and `fstat` reference the same inode β€” mismatch indicates a symlink race between opening and checking\n\nBoth operations reject any symlink detection as a security violation. On success, `ReadFileSafely` reads all content via buffered I/O; on success, `WriteFileSafely` truncates to zero first (safe only because symlinks were ruled out), then writes the provided content.\n\n#### Error Handling Contract\n\n| Function | Wrapped Errors (`%w`) | Swallowed | Sentinel Unwrapped |\n|---|---|---|---|\n| `ReadFileSafely` | Yes, for every I/O error | No | Symlink race check |\n| `WriteFileSafely` | Yes, for every I/O error | No | Symlink race check |\n| `LoadGitignore` | No | Missing file β†’ `(nil, nil)` | None |\n| `ShouldIgnoreFile` | No | `SafeResolve` err β†’ returns `true` | None |\n| `DiscoverCodeFiles` | No | Extensive β€” walk callback errors swallowed at each node | None |\n| `IsBinaryFile` | No | All open/read errors β†’ `false` | None |\n\n#### File Discovery Algorithm (Multi-Layer Ignore Rules)\n\nA file is considered \"ignored\" if it matches **any** of these conditions:\n\n- Matches a compiled gitignore pattern list\n- Has a path component starting with `.` or ending in `.egg-info`\n- Has an extension matching the user-provided ignored extensions list (case-insensitive, supports both `.` prefix and suffix forms)\n- Is detected as binary by reading up to 1024 bytes and scanning for null byte (`\\x00`)\n\nIf any single check returns true, the file is excluded. This is a **union** of ignore conditions.\n\nThe primary business rule is to recursively walk the repository root and discover high-signal source code files while filtering out noise: build artifacts, dependency directories, output files, dot-prefixed hidden paths, `.egg-info` markers, binary files, and any user-defined ignore patterns (via gitignore compilation).\n\n#### Binary Detection Rule\n\nA file is considered binary if its first 1024 bytes contain a null byte (`\\x00`). Files that cannot be opened are treated as non-binary (returns false), avoiding the loading of entire files into memory for classification.\n\n### `git_tools.go` β€” Git Process Abstraction\n\n#### Core Functions\n\n| Function | Input | Output |\n|---|---|---|\n| [`RunGit`](file:///path/to/git_tools.go#L15-L64) | `repoRoot string`, variadic `args ...string` | `(string, error)` |\n| [`VerifyGitRepo`](file:///path/to/git_tools.go#L70-L89) | `repoRoot string` | `error` |\n\n#### Execution Model\n\nBoth functions spawn the `git` binary via `exec.Command(\"git\", ...)` with `--no-pager`, restricting execution to the provided root directory. Both capture stdout and stderr into buffers synchronously. No standard library packages beyond `bytes`, `fmt`, `os/exec`, `strings` are imported.\n\n#### Error Handling Contract\n\n| Function | Failure Path | Behavior |\n|---|---|---|\n| `RunGit` | Exit code β‰  0 | Wraps error: `git command failed: %v, stderr: %s`. Returns `(string, error)` with stdout and wrapped error. No panic", - "internal/config": "# `internal/config` β€” Package Documentation\n\n## Responsibility and Data Flow\n\nThe package implements a **multi-layer configuration resolution** pipeline for the code analysis tool. Configuration values flow through four precedence tiers: hard-coded defaults β†’ YAML file (`~/.config/code-reducer.yaml`) β†’ environment variables β†’ CLI flags. Each layer overwrites prior values when present; missing layers are transparently skipped.\n\nThree properties are resolved independently: `ModelID`, `OllamaBaseURL`, and `OllamaNumCtx`. The package also structures the analysis pipeline into named **extraction steps** (`ExtractionStep`), each carrying a distinct prompt template. These steps can be customized per project via YAML or wholesale replaced by CLI flag.\n\nAll non-structural state (resolved config, local variables) is ephemeralβ€”scoped to function calls and returned as values. No package-level mutable state survives beyond initialization. Concurrency primitives are absent; the package is single-threaded in its design.\n\n## Types\n\n### `ExtractionStep`\n\n```go\ntype ExtractionStep struct {\n Name string // step identifier (e.g., \"api-signatures\", \"business-logic\")\n Prompt string // prompt template applied to this phase during analysis\n}\n```\n\nInstances are held in a package-level slice (`DefaultExtractionSteps`). The slice is immutable after declaration; callers receive copies or slices of it.\n\n### `Config`\n\n```go\ntype Config struct {\n ModelID string // resolved model identifier (last-wins: default β†’ YAML β†’ env β†’ flag)\n OllamaBaseURL string // resolved base URL (default β†’ YAML β†’ env; CLI flag omitted)\n OllamaNumCtx int // resolved context size, validated positive (default β†’ YAML β†’ env β†’ flag)\n DocsDir string // path to the documents directory under analysis\n ExtractionSteps []ExtractionStep // ordered list of extraction phases\n Ignore []string // absolute paths excluded from traversal\n IgnoreExtensions []string // file extensions excluded from traversal\n}\n```\n\n## Constants and Defaults\n\n### Environment Variable Keys\n\n| Constant | Value | Purpose |\n|---|---|---|\n| `CodeReducerModelIdEnvKey` | `\"CODE_REDUCER_MODEL_ID\"` | Overrides `Config.ModelID` when set. |\n| `OllamaBaseUrlEnvKey` | `\"OLLAMA_BASE_URL\"` | Overrides `Config.OllamaBaseURL` when set. |\n| `OllamaNumCtxEnvKey` | `\"OLLAMA_NUM_CTX\"` | Overrides `Config.OllamaNumCtx` when set; value must parse as a positive integer. |\n\n### Default Values\n\n| Constant | Value | Purpose |\n|---|---|---|\n| `OllamaDefaultBaseURL` | `\"http://localhost:11434\"` | Fallback base URL when no prior layer supplies one. |\n| `OllamaDefaultModelID` | `\"ornith:9b\"` | Default model identifier. |\n| `OllamaDefaultNumCtx` | `8192` | Default context size. |\n\n### Configuration File\n\n| Constant | Value | Purpose |\n|---|---|---|\n| `ConfigFileName` | `\".code-reducer.yaml\"` | Filename resolved relative to the current working directory by `getConfigPath`. |\n\n## Package-Level Immutable Defaults\n\nThree slices are declared at package init and never mutated:\n\n```go\nvar DefaultIgnores []string // system-supplied ignore paths\nvar DefaultIgnoredExtensions []string // system-supplied ignored extensions\nvar DefaultExtractionSteps []ExtractionStep // ordered extraction phases\n```\n\nCallers may append to these slices externally, but the package itself never reassigns them. `SaveConfig` writes the current slice state back to disk; callers must mutate before save if they need custom defaults.\n\n## Public API Surface\n\n### Path Resolution and Existence Check\n\n#### `getConfigPath(cwd string) (string, error)`\n\nConstructs the full path to `.code-reducer.yaml` by joining `cwd` with `ConfigFileName`. Returns a non-nil error only on I/O failure during construction; normal cases return `nil`.\n\n#### `ConfigExists(cwd string) bool`\n\nReturns whether the configuration file exists at the resolved path. Internally calls `os.Stat`; all errors (permission denied, I/O failures, etc.) are swallowed and treated as \"does not exist\". The function returns no error.\n\n### Configuration Loading\n\n#### `LoadConfig(cwd string) (*Config, error)`\n\nReads and parses the YAML configuration file at the resolved path. Error handling:\n- File-read failure (`os.ReadFile`) β†’ returned to caller unchanged (pass-through).\n- YAML parse failure β†’ wrapped as `\"failed to parse yaml config: %w\"`.\n\nReturns a non-nil `*Config` on success; the struct is initialized with zero values before parsing, so callers receive a populated object even when the file contains only valid empty mappings.\n\n#### `ResolveConfig(repoRoot, modelIdFlag, numCtxFlag string) (*Config, error)`\n\nTop-level entry point for full configuration resolution. Algorithm:\n1. Calls `LoadConfig` to read the YAML file. If the error is a non-existent-file sentinel (`os.IsNotExist(err)`), it swallows and initializes an empty `Config{}`; all other errors propagate wrapped as `\"failed to load configuration file:\"`.\n2. Layers defaults (hard-coded constants) over the loaded config.\n3. Layers environment variable overrides using `os.Getenv` for `CODE_REDUCER_MODEL_ID`, `OLLAMA_BASE_URL`, and `OLLAMA_NUM_CTX`. Non-empty values are used; empty strings fall through. Env read errors are swallowed.\n4. Applies CLI flags (`modelIdFlag`, `numCtxFlag`). The flag value is parsed via `strconv.Atoi`; only `err == nil \u0026\u0026 n \u003e 0` is acceptedβ€”invalid or non-positive values cause the prior layer's value to be retained silently.\n\nReturns a fully resolved `*Config`. Errors are either pass-through (from `LoadConfig`) or wrapped with `\"failed to load configuration file:\"`. Notable: if YAML is malformed, the parse error wraps and propagates; otherwise resolution always succeeds for missing files.\n\n### Configuration Persistence\n\n#### `SaveConfig(cwd string, cfg *Config) error`\n\nSerializes the config struct back to disk using `os.WriteFile` with `0600` permissions. Error handling:\n- Write success β†’ returns `nil`.\n- Parse failure (marshal YAML) β†’ wrapped as `\"failed to marshal yaml: %w\"`.\n- File-write failure β†’ wrapped as `\"failed to write config file: %w\"`.\n\nWrites directly (no temp-file + rename); a partial write can leave a truncated file on disk. Permission normalization is implicitβ€”existing files with different permissions will fail the write.\n\n## Utility Functions\n\n### `MergeAndDeduplicate[T comparable](a, b []T) []T`\n\nGeneric slice merge that deduplicates entries. Merges `b` after `a`; duplicates are removed via a map lookup and never reported as errors. Returns a new slice; the original inputs remain unmodified. The function is safe for concurrent use (no shared mutable state).", - "internal/engine": "# Module: internal/engine β€” Code Documentation Pipeline Engine\n\n## Overview\n\nThe `internal/engine` package implements a Map-Reduce pipeline that generates and maintains technical documentation for a Go codebase using Ollama-compatible LLM inference endpoints. The engine produces three artifacts per run: `architecture.md` (global overview), `quickstart.md` (onboarding guide), and per-module API summaries under `modules/`. Execution is gated by a repository-level lock to prevent concurrent runs against the same root, and state persistence across invocations is handled by an on-disk metadata cache.\n\n**Data flow:** Repository isolation (`security.EnsureGitignoreHasLockfile`) β†’ lock acquisition (`security.AcquireLock`) β†’ LLM client construction from config β†’ pipeline dispatch (`RunInit` or `RunUpdate`) β†’ recursive tree synthesis with bottom-up reduction β†’ standard docs generation β†’ final state persistence. Errors propagate to the caller unless explicitly swallowed and logged via the `Event{Type: \"status\", Message: ...}` channel.\n\n---\n\n## Pipeline Orchestration\n\n### Runner Entry Point (`runner.go` β€” `Runner.Run`)\n\nThe public entry point for all pipeline invocations. Responsibilities: repository isolation, concurrency guard acquisition, LLM client construction from config, and mode-gated dispatch to the appropriate pipeline method.\n\n```\nRun(ctx, repoRoot, mode string, onEvent func(Event)) error\n β”œβ”€ ensure .gitignore lockfile entry (best-effort; failure logged as warning)\n β”œβ”€ acquire repository-level lock via security.AcquireLock(repoRoot)\n β”‚ └─ defer lock.Unlock() β€” released on any return path including errors\n β”œβ”€ build *LLMClient from r.cfg.ModelID, BaseURL, NumCtx\n β”œβ”€ switch mode:\n β”‚ case \"init\": client.RunInit(ctx, repoRoot, r.cfg, onEvent)\n β”‚ case \"update\": client.RunUpdate(ctx, repoRoot, r.cfg, onEvent)\n β”‚ default: return fmt.Errorf(\"unsupported mode: %s\", mode)\n```\n\nThe `cfg *config.Config` field is read-only within this file; all mutations to the runner's configuration originate before construction. The lock returned by `security.AcquireLock(repoRoot)` is a local variable with deferred cleanupβ€”its internal type and mutual exclusion semantics are not observable from this source alone.\n\n### Orchestrator Pipelines (`orchestrator.go` β€” `RunInit`, `RunUpdate`)\n\n#### Full Run (`RunInit`)\n\nExecutes an all-or-nothing regeneration regardless of cache state:\n\n1. **Setup** β€” Load `.gitignore` via `tools.LoadGitignore(repoRoot)`; merge with user ignore list; load metadata cache from disk via `cache.loadMetadataCache()`. Failures logged as warnings, not returned.\n2. **Map Phase** β€” Discover code files via `tools.DiscoverCodeFiles(...)`, compute SHA-256 per file via `computeSHA256(repoRoot, f)`, build directory tree via `buildTree(codeFiles)`. Hash computation errors silently skipped in loops; only stored if hash computed successfully.\n3. **Mark All Affected** β€” Recursively flag every node as affected (init starts fresh).\n4. **Reduce Phase** β€” Call `synthesizeNode` bottom-up on the tree to produce a root summary string and per-module summaries via LLM calls through `c.CallLLM`.\n5. **Generate Standard Docs** β€” Produce `architecture.md` and `quickstart.md` from the root summary via two additional `CallLLM` invocations: one for architecture, one for quickstart. Writes via `tools.WriteFileSafely`; errors wrapped with context (e.g., `\"failed to write quickstart.md: %w\"`).\n6. **Update AGENTS.md** β€” Read existing file via `tools.ReadFileSafely(repoRoot, \"AGENTS.md\")`. If missing, create with guidelines header. If present but does not contain \"AI Agent Guidelines\", append separator + new content block. Only writes when content actually differs. Errors: if read fails but write succeeds β†’ `\"failed to write AGENTS.md: %w\"`; if both fail β†’ wrapped errors chain.\n7. **Teardown** β€” Persist metadata cache via `saveMetadataCache(repoRoot, docsDir, cache)`. Failure logged as warning, function always returns nil at end regardless of this failure.\n\n#### Incremental Run (`RunUpdate`)\n\nExecutes only re-generations for directories whose contents actually changed:\n\n1. **Setup** β€” Load pipeline state from metadata cache (same path as init).\n2. **Map Phase** β€” Discover code files, hash each via `computeSHA256`, build tree.\n3. **Change Detection** β€” Compare current file hashes against cached hashes in `cache.Files`:\n - New file in codebase but absent from cache β†’ `FileChange{Status: \"Added\"}`\n - Existing file with different hash β†’ `FileChange{Status: \"Modified\"}`\n - Cached file no longer in codebase β†’ `FileChange{Status: \"Deleted\"}` (also removed from cache)\n4. **Prune Stale Modules** β€” Remove any module summaries whose directory is no longer active, delete their markdown files on disk via `os.Remove(absModuleFile)` (error assigned to `_`, never logged).\n5. **Propagate Affected** β€” Run tree-aware propagation (`propagateAffected`/`determineAffected`) to mark directories affected by changes. Errors from these calls are discarded without capture.\n6. **Early Exit** β€” If zero directories are affected β†’ pipeline ends with \"up to date\" status, no further synthesis occurs.\n7. **Conditional Reduce** β€” If root-level change detected or global docs don't exist on disk (`os.Stat(absArch)`/`os.Stat(absQs)`), run full synthesis; otherwise skip and reuse existing `architecture.md`/`quickstart.md`.\n8. **Teardown** β€” Save updated cache via `saveMetadataCache(repoRoot, docsDir, cache)`.\n\n### Directory Tree Construction \u0026 Affected Propagation (`tree.go`)\n\n#### Type Definitions\n\n| Type | Fields | Purpose |\n|------|--------|---------|\n| `FileChange` | `Path string`, `Status string` | Models a file operation with path and status (`Added`, `Modified`, `Deleted`). Consumed by `determineAffected`. |\n| `DirNode` | `Path string`, `Files []string`, `Children map[string]*DirNode` | Hierarchical tree node holding files at that level and child directories in its subtree. Produced by `buildTree`. |\n\n#### Functions (all unexported)\n\n- **`buildTree(codeFiles)`** β€” Converts a flat list of file paths into nested `*DirNode` structure, preserving directory nesting by splitting on `/`. Appends to `DirNode.Files`; populates `DirNode.Children` via map assignment.\n- **`propagateAffected(tree, changeset)`** β€” Traverses the tree recursively: if any direct child of the current node is in the changeset, mark it affected. Accumulates affected status upward through ancestors. Returns mutated `affectedDirs`.\n- **`determineAffected(tree, changeset)`** β€” Computes module path for each directory via `ToSafeMarkdownFilename`, checks existence on disk via `os.Stat(absModulePath)`, reads metadata cache (`cache.Modules[n.Path] == \"\"`), marks node affected if any check fails. Only `os.IsNotExist` is acted on; all other errors are silently dropped.\n\n---\n\n## LLM Client Abstraction (`client.go`)\n\n### Type: `LLMClient`\n\n```go\ntype LLMMClient struct {\n ModelID string\n BaseURL string\n NumCtx int\n HTTPClient *http.Client // default transport, no custom config (no proxy/redirect/TLS)\n}\n```\n\nConstructed via `NewLLMClient(modelID, baseURL, numCtx)` with a 10-minute HTTP timeout. Fields set once; never reassigned within any method.\n\n### Methods on `*LLMClient`\n\n#### `CallLLM(ctx context.Context, systemPrompt string, messages []Message, jsonFormat bool) (string, error)`\n\nFull lifecycle: prepends system prompt as first message with role `\"system\"` β†’ serializes into Ollama `/api/chat` JSON payload with model ID and context window size β†’ non-streaming HTTP POST β†’ response body fully read on 200 OK or truncated at 1 KB via `io.LimitReader(resp.Body, 1024)` on non-OK status. Returns only assistant content string; returns empty string on failure.\n\n**Error behavior:**\n| Failure path | Error type | Wrapping strategy |\n|---|---|---|\n| JSON marshal fails in request prep | `fmt.Errorf(\"failed to marshal request: %w\", err)` | Wrapped with `%w` |\n| `http.NewRequestWithContext` fails | raw error from `NewRequestWithContext` | **Not wrapped** β€” passed through as-is |\n| HTTP Do (network-level failure) | raw error from `c.HTTPClient.Do(req)` | **Not wrapped** |\n| Body read fails on 200 OK path | raw error from `io.ReadAll` | **Not wrapped** |\n| JSON unmarshal fails on response | `fmt.Errorf(\"failed to parse response: %w\", err)` | Wrapped with `%w` |\n| Non-OK status (body truncated at 1 KB) | `fmt.Errorf(\"ollama api error: status %d, response: %s\", ...)` | New error constructed; body discarded if read fails |\n\n#### `GetBaseSystemPrompt() string`\n\nReturns the fixed base role definition and defensive guidelines inherited by every LLM call regardless of task type.\n\n#### `GetDefaultSystemPrompt(command string) string`\n\nAppends command-specific instructions based on the command string: two named variants exist (`module_synthesis`, `architecture`); any other value receives only the base layer.\n\n---\n\n## Response Parsing Layer (`json_parser.go`)\n\n### Functions (all unexported except `StripOuterMarkdownFence`, `CleanJSONResponse`, `UnmarshalJSONResponse`)\n\nInternal helper `extractBalancedJSON` is lowercase and excluded from public surface area.\n\n#### Business Rules\n\n1. Markdown code fences (` ``` `) surrounding JSON must be stripped before parsing β€” handled by `StripOuterMarkdownFence`.\n2. Balanced bracket matching required β€” strings containing `{`, `[`, `}`, `]` literals (including escaped characters like `\\n`) must not break extraction logic β€” handled by internal rune-level scanner with escape state tracking inside string literals.\n3. When multiple balanced JSON candidates exist in a single response, each must be attempted individually until one succeeds, or the entire trimmed string is tried as a final fallback β€” handled by `CleanJSONResponse`/`UnmarshalJSONResponse`.\n\n#### Functions\n\n| Function | Behavior | Error surface | Swallowed? |\n|---|---|---|---|\n| `StripOuterMarkdownFence(s string) string` | Strips surrounding markdown fences; falls back to trimmed input if none match | None β€” returns string only | N/A (no error return) |\n| `CleanJSONResponse(s string) string` | Iterates stripped content, calls `extractBalancedJSON`, returns first successful candidate or original trimmed input unchanged | None β€” returns string only | Yes β€” unbalanced JSON inside response is swallowed silently |\n| `UnmarshalJSONResponse(s string, target interface{}) error` | Multi-layered attempt: (1) collect balanced candidates, try each into target; keep last error in `lastErr`; (2) if any succeeded, return nil; otherwise try entire trimmed string; (3) terminal errors β€” `\"failed to unmarshal JSON candidates: %w\"` or `\"no valid JSON found in response\"` | Explicit (`error`) | Partial parse failures collected and re-emitted via `%w` on first path; \"no valid JSON\" covers second path |\n\n---\n\n## Synthesis Engine (`synthesize.go`, `chunking.go`)\n\n### Context-Aware Chunking (`chunking.go` β€” all unexported)\n\nThe system processes extracted code/fact items in batches constrained by LLM context limits (`NumCtx * 3`). Three distinct processing modes exist:\n\n#### Module Synthesis (`reduceInChunks`)\n\nSynthesizes architectural-level nodes into architecture descriptions using a `\"module_synthesis\"` system prompt. Each batch represents evidence contributing to understanding the module's design intent. Response passes through `StripOuterMarkdownFence`. Errors wrapped with `\"LLM error during synthesis: %w\"`.\n\n#### File Fact Consolidation (`reduceFileFacts`)\n\nConsolidates extracted facts about a specific step within a file with deduplication and merging. System prompt built locally by concatenating `c.GetBaseSystemPrompt()` + fixed role string. Response also passes through `StripOuterMarkdownFence`. Errors wrapped with `\"LLM error during file fact consolidation: %w\"`.\n\n#### Recursive Convergence (`reduceItems`)\n\nCore business rule: reduce items in batches β†’ collect intermediate results β†’ recursively apply reduction until everything converges into a single output string. Multi-pass approach ensures information is progressively synthesized rather than lost across multiple LLM calls. Termination guaranteed when all data fits within one batch (short-circuits at top with `len(batches) == 1`). Recursion terminates because:\n1. Single-batch case short-circuits at top (`len(batches) == 1`)\n2. Recursive step processes each batch through itself again with same `maxChars`\n\n**Graceful Truncation Policy:** When content exceeds the context window, single oversized items get truncated with `\\n...[truncated]` marker appended; when multiple items exceed limits collectively, each gets capped at `maxChars / len(items)` characters before being processed as one batch. Infinite-loop guard: if batch size equals item count for all items (can't merge), falls back to per-item truncation using `allowedPerItem = maxChars / len(items)`. Still silent β€” no error returned.\n\n#### Overlapping Chunk Support (`chunkTextWithOverlap`)\n\nUtility for splitting raw text into overlapping chunks to maintain context continuity between adjacent segmentsβ€”useful when feeding source material directly into LLMs without pre-processing through the chunking logic. No I/O. Silent swallow: `maxRunes \u003c= 0` β†’ returns full text as single chunk; `overlapRunes \u003e= maxRunes` β†’ clamps to half without warning; final partial chunk reaching end of runes β†’ breaks loop with no error or truncation marker (unlike `reduceItems`).\n\n### Hierarchical Module Synthesis (`synthesize.go` β€” `synthesizeNode`, all unexported)\n\nPerforms recursive synthesis of directory structures into hierarchical summariesβ€”transforming raw source code and subdirectory metadata into consolidated documentation at each level of a module tree.\n\n**Algorithmic Flow:**\n1. **Context Check β†’ Cache Reuse Gate**: If current node not in `affectedDirs` AND cached module summary exists, return immediately without processing. Short-circuits unaffected directories.\n2. **Recursive Child Synthesis (Bottom-Up)**: Sort children alphabetically, recursively call `synthesizeNode` on each child directory; results collected into map keyed by child name.\n3. **File Processing Loop**: For each file in current node: hash-first cache lookup (if precalculated hash exists and matches cached SHA256 β†’ reuse cached facts without reading disk); read + hash fallback otherwise (read via `tools.ReadFileSafely`, compute SHA256, update cache entry if already existed with that hash); extraction pipeline on cache miss (chunk content using dynamic limit derived from `c.NumCtx Γ— 4 characters Γ— 0.75` with minimum of 512 tokens and max overlap cap; for each chunk iterate over every step in `cfg.ExtractionSteps`, sending to LLM with system prompt built from base + step-specific prompt; consolidate all chunks for that step via `reduceFileFacts`; append consolidated result under step's name heading).\n4. **Component Assembly**: After processing files, append any non-empty child summaries to components list. If no components exist at all (no files AND no children), clear module cache and return empty string.\n5. **Final Synthesis \u0026 Persistence**: Pass assembled components to `reduceInChunks` for directory path; store result in both memory cache (`cache.Modules[node.Path]`) and on disk at `\u003crepoRoot\u003e/\u003cdocsDir\u003e/modules/\u003csafe-filename\u003e` via `tools.WriteFileSafely`; return final summary string.\n\n**Domain concepts:** Hierarchical module synthesis (bottom-up: files contribute facts, subdirectories contribute their own summaries, parent combines both); context-aware truncation (file content limits scale with `c.NumCtx`, reserving 75% of tokens for file data and 25% for prompts/output); two-level caching (files cache individual facts keyed by SHA256, modules cache full synthesized summaries keyed by path; reuse only when node not affected); multi-step extraction pipeline (each file passes through configurable sequence of LLM steps, each producing independent fact set consolidated before combining with other files' results).\n\n**Error handling in `synthesizeNode`:**\n| Source | Pattern | Propagation |\n|---|---|---|\n| Context cancel | Direct return at top and inside per-file loop (`if err := ctx.Err(); err != nil { return \"\", err }`) | Upward, no wrapping |\n| LLM call failure | Wrapped: `\"LLM error extracting %s for %s: %w\"` with step name + file path | Upward |\n| File read failure | Logged via `logEvent(\"status\", ...)`, skipped silently; no hash/facts/anything added to components | None (swallowed) |\n| Module write failure | Wrapped: `\"failed to write module documentation for %s: %w\"` with dir path | Upward |\n| Recursive child error | Propagated as-is (`return \"\", err`) | Upward |\n| Empty components | Returns `(\"\", nil)`, clears `cache.Modules[node.Path] = \"\"` | None (clean return) |\n\n---\n\n## Filesystem State Management (`cache.go`)\n\n### Type: `MetadataCache`\n\n```go\ntype MetadataCache struct {\n Files map[string]FileCacheEntry // virtual path β†’ cached SHA256 + facts\n Modules map[string]string // module identifier β†’ summary string\n}\n```\n\nBoth maps initialized with `make()` when nil; populated during load/save lifecycle; serialized to disk. No in-memory locking applied within this file.\n\n### Functions (all unexported except `IsInitialized`)\n\n#### Core Concept: Filesystem-Based State Cache\n\nPersists an in-memory cache to disk so expensive computations can be avoided across runs. Stored as `docsDir/.metadata.json`, holds two pieces of state:\n- **File-level metadata** (`Files map`): maps virtual paths (filenames) to cached SHA256 hashes plus associated \"facts\" strings.\n- **Module-level metadata** (`Modules map`): maps module identifiers to string values.\n\n#### Functions\n\n| Function | Direction | Path constructed | Notes |\n|---|---|---|---|\n| `loadMetadataCache()` | Read | `docsDir/.metadata.json` | Attempts to read JSON; if absent, returns freshly initialized empty cache (not error); if present but unparseable, errors. Null maps normalized to empty on load. |\n| `IsInitialized(repoRoot, docsDir string) bool` | Read (existence probe) | `docsDir/.metadata.json` | Probes for existence of `.metadata.json`. Successful read β†’ true; any non-nil error β†’ false. N/A for missing file (returns false). |\n| `saveMetadataCache()` | Write | `docsDir/.metadata.json` | Serializes struct back to JSON with indentation, writes atomically via `tools.WriteFileSafely`. Returns raw errors from marshal/write unchanged. |\n| `computeSHA256(virtualPath)` | Read (virtual path) | caller-supplied virtual path | Reads file by virtual path, runs SHA-256 on content, returns hex-encoded digest. No disk I/O within this file; pure in-memory (`crypto/sha256`). |\n\n**Error handling patterns:**\n| Function | On missing cache file | On generic read/write error | On JSON parse error |\n|---|---|---|---|\n| `loadMetadataCache` | Returns empty initialized `*MetadataCache`, nil error | Wraps with `\"failed to read metadata cache: %w\"` | Wraps with `\"failed to unmarshal metadata cache: %w\"` |\n| `IsInitialized` | N/A (returns false) | N/A β€” any non-nil error β†’ false | β€” |\n| `saveMetadataCache` | β€” | Returns raw error from `tools.WriteFileSafely` | Returns raw error from `json.MarshalIndent` |\n| `computeSHA256` | β€” | Returns raw error from `tools.ReadFileSafely` | N/A (no parsing) |\n\n---\n\n## Utility Functions (`utils.go`)\n\n### Function: `ToSafeMarkdownFilename(modulePath string) string`\n\nPure string transformation with no business logic or complex algorithmic flow. Maps a module path to a safe filename for markdown documentation.\n\n**Conversion pipeline:**\n1. Replace every `/` character in input `modulePath` with `_`.\n2. If result is empty or equals `\".\"`, substitute with `\"root\"`.\n3. Append `.md` and return.\n\n**Domain concept:** Module-to-document routing β€” given a module identifier, produce corresponding markdown file path. Assumes caller already owns consistent `modulePath` convention; this utility only adapts that path for filesystem-safe filenames.\n\n---\n\n## Concurrency \u0026 State Summary\n\n### Mutable State Across Files\n\n| Variable | File | Mutated By | Description |\n|---|---|---|---|\n| `MetadataCache.Files` | `cache.go` | load/save lifecycle | Map[string]FileCacheEntry β€” populated during load/save; serialized to disk |\n| `MetadataCache.Modules` | `cache.go` | load/save lifecycle | Map[string]string β€” same pattern as Files |\n| `DirNode.Files` | `tree.go` | buildTree only | Appended in tree construction, read-only otherwise |\n| `DirNode.Children` | `tree.go` | buildTree only | Populated via map assignment during tree construction |\n| `affectedDirs` (map[string]bool) | `orchestrator.go`, `tree.go` | Recursive closures: markAllAffected, collectDirs, determineAffected, propagateAffected | Populated by recursive traversal of tree nodes; passed to synthesizeNode. Local scope only in each invocation. |\n| `cache.Files` / `cache.Modules` (local) | `orchestrator.go`, `synthesize.go` | synthesizeNode β†’ updateCacheFromHashes, buildModuleMap, etc. | Populated in RunInit; modified during synthesis in RunUpdate. Local to function invocations; no shared mutable state across goroutines visible here. |\n| `cfg *config.Config` (field on Runner) | `runner.go` | Set once in NewRunner; not mutated within this file | Read through methods delegated to other packages only. External mutation possible before construction, but no writes occur here. |\n\n### Concurrency Mechanisms: None Detected\n\nNo `sync.Mutex`, channels (`chan`), atomic operations, or other synchronization primitives appear in any of these files. The recursion in `synthesizeNode` is itself a concurrency primitive (caller may invoke from many goroutines), so map writes to `cache.Files` and `cache.Modules` are unprotected against concurrent access if called with multiple goroutinesβ€”but no such protection exists within this codebase's visible scope.\n\n### External Communication Summary\n\n| File | I/O Category | Targets |\n|---|---|---|\n| `client.go` | Network (HTTP POST) | Ollama-compatible endpoint at `{BaseURL}/api/chat`, 10-minute timeout, non-streaming |\n| `json_parser.go` | None | Pure in-memory string/byte manipulation; only imports from standard library (`encoding/json`, `fmt`, `regexp`, `strings`) |\n| `cache.go` | Disk (read/write) | `docsDir/.metadata.json` via `tools.ReadFileSafely` / `tools.WriteFileSafely` |\n| `orchestrator.go` | Network + Disk | LLM service calls, local filesystem under `repoRoot` (md files, cache, dir tree), `.gitignore` read/write |\n| `runner.go` | Disk write (lockfile) | `.gitignore` modification for lockfile entry; lock file creation/update via `security.AcquireLock`; defer cleanup on any return path |\n| `synthesize.go` | Network + Disk | LLM calls per chunk per extraction step, file reads via `tools.ReadFileSafely`, final writes via `tools.WriteFileSafely` with error wrapping |\n| `tree.go` | Disk read (one op) | `os.Stat(absModulePath)` for module existence check; all other operations are pure tree construction/traversal |\n| `utils.go` | None | Pure string transformation, no external I/O whatsoever |", - "internal/security": "# `internal/security` β€” Package Documentation\n\n## Module Responsibility \u0026 Data Flow\n\nThe `security` package encapsulates three orthogonal concerns for code-reducer processes: **sandbox enforcement** (preventing path traversal beyond the repository boundary), **process-level mutual exclusion** via file-based locks, and **operational hygiene** (keeping lock state out of version control). No networking, database access, or external process communication occurs within this package.\n\nAll functions are pure-Go with no global mutable state. Every observable side effect is local to a single `*SimpleLock` receiver instance or the calling goroutine's stack. Errors follow a strict wrapping convention: security-critical violations and actionable sentinel messages return unwrapped plain strings; standard-library errors use `%w`; best-effort cleanup paths swallow non-fatal I/O failures silently.\n\n**Data flow** proceeds in two independent tracks that share no state except through the repository root path string:\n\n1. **Path resolution track:** `SafeResolve` β†’ called by `AcquireLock` to canonicalize the lock file location before attempting acquisition.\n2. **Lock lifecycle track:** `AcquireLock` β†’ returns a `*SimpleLock` receiver β†’ caller uses it for work β†’ calls `Unlock()` for release β†’ `EnsureGitignoreHasLockfile` runs during cleanup to append the lock filename to `.gitignore`.\n\n---\n\n## Public API Surface\n\n### Types\n\n#### `SimpleLock`\n\nFile-based process lock with internal mutex protection. The struct is constructed by `AcquireLock`; once acquired, `lockPath` and `file` are set exactly once under mutex scope, then only modified during `Unlock()`.\n\n| Field | Type | Semantics |\n|-------|------|-----------|\n| `lockPath` | `string` | Absolute path to the `.code-reducer.lock` file. Set in constructor; zero-valued if acquisition failed or was never called. |\n| `file` | `*os.File` | Opened lock file handle. Acquired with `O_CREATE\\|O_EXCL`; set to `nil` inside `Unlock()`. |\n| `mu` | `sync.Mutex` | Per-instance mutex protecting `closed` and the pointer-clear of `file` during unlock. |\n| `closed` | `bool` | Transition flag: `false` while locked, flips to `true` once `Unlock()` completes its write operations (under mutex). |\n\n#### Functions\n\n##### `SafeResolve(repoRoot, inputPath string) (string, error)`\n\nComputes the absolute path of an input relative to a repository root. The resolution is symlink-aware: ancestor components are evaluated through their physical targets via `filepath.EvalSymlinks` before being walked upward until one succeeds. The final result must lie strictly inside the resolved rootβ€”any escape beyond it returns an unwrapped string error indicating a security violation.\n\n##### `AcquireLock(repoRoot string) (*SimpleLock, error)`\n\nResolves the lock file path through `SafeResolve`, then attempts atomic creation with `O_CREATE\\|O_EXCL`. On success, writes the process PID to the newly created file and returns a populated `*SimpleLock`. If the file already exists (anewer process holds it), returns an unwrapped sentinel string describing that another process has acquired the lock.\n\n##### `EnsureGitignoreHasLockfile(repoRoot string) error`\n\nReads `.gitignore`; if present, appends `# Code-Reducer Lockfile\\n.code-reducer.lock\\n` only when not already contained in the file. If `.gitignore` does not exist, creates it with append semantics (`O_APPEND\\|O_CREATE\\|O_WRONLY`). Errors from reading or writing are wrapped; non-existence of `.gitignore` is treated as expected and returned as `nil`.\n\n#### Methods\n\n##### `(l *SimpleLock).Unlock() error`\n\nIdempotent release: acquires the internal mutex, closes the file handle (discarding any close-error), removes the lock file from disk via best-effort `os.Remove`, sets `closed = true`, then releases the mutex. A second call is safe and returns `nil`. Swallowed errors for cleanup operations are discarded; removal failures that occur after a successful close surface as wrapped errors only in the specific narrow case where both close-succeeded-and-remove-failedβ€”but this path effectively swallows non-`IsNotExist` remove errors in practice.\n\n---\n\n## Business Rules \u0026 Domain Concepts\n\n### Repository Boundary Enforcement\n\nAll internal paths must resolve strictly within `repoRoot`. Any resolution that escapes beyond it returns a standalone string error: `\"security violation: path traversal detected: %q\"`, never wrapped. This is the one unwrapped class of non-sentinel errors in the packageβ€”security-critical, so callers detect it via exact match or `strings.Contains` rather than type assertion.\n\n### Symlink-Aware Path Resolution\n\nAncestors are evaluated through their physical targets before path reconstruction. This prevents traversal attacks that use symlinked directories to bypass the sandbox while preserving logical input structure for legitimate uses. Implementation: `filepath.EvalSymlinks(absRoot)` resolves the root once; subsequent ancestor walks use `os.Lstat` with a fallback to `EvalSymlinks` on each component until one exists, then rebuild upward from that point.\n\n### Process-Level File Locking\n\nThe lock file is `.code-reducer.lock`. Acquisition uses `O_EXCL` for atomic creationβ€”no two processes can hold the lock simultaneously on the same filesystem. On success, the PID of the acquiring process is written as a single-line string (`fmt.Sprintf(\"%d\\n\", os.Getpid())`). The lockfile content therefore serves as both a record and a mechanism for stale detection: if another process exits without releasing its lock, a new acquisition attempt detects the existing file and reports an actionable error instructing manual cleanup.\n\n### Stale Lock Detection\n\nIntentional behaviorβ€”the system does not attempt to detect or kill the holding process. The caller receives an unwrapped string describing that the lock is held by another process and must clean up manually. This keeps the package non-invasive; it never interacts with external processes.\n\n### Automated Gitignore Integration\n\nThe lockfile path is appended to `.gitignore` during unlock so it does not enter version control. Idempotent: if a line containing `code-reducer.lock` already exists, no duplicate entry is written. The lock filename itself is the only thing tracked; comment prefix provides human readability without affecting git behavior.\n\n---\n\n## Mutable State \u0026 Concurrency Analysis\n\n### Per-Instance State\n\n| Field | Mutated By | Notes |\n|-------|-----------|--------|\n| `lockPath` | Constructor (`AcquireLock`) | Set exactly once. Zero value if acquisition failed or never called. Not mutated after construction. |\n| `file` (*os.File) | Constructor, then `Unlock()` | Pointer cleared inside the mutex during unlock so subsequent reads yield nil. |\n| `mu` (sync.Mutex) | All methods on receiver | Acquired/released by every method; not modified as a value. |\n| `closed` (bool) | `Unlock()` only | Flips from `false` β†’ `true` under mutex. Once written, stays true for the lifetime of the receiver. |\n\n### Concurrency Protection\n\nThe `sync.Mutex` protects all writes to `l.closed` and the pointer-clear of `l.file` inside `Unlock()`. No other concurrent access paths touch these fields in this fileβ€”`AcquireLock` constructs a new instance, so no shared writer exists for an already-acquired lock. The mutex serializes unlock calls on a single receiver only; it is not a global contention point across processes.\n\n---\n\n## Side Effects \u0026 I/O Communication\n\n### Filesystem Operations by Function\n\n| Function | Reads | Writes | Deletes/Atomic | Notes |\n|----------|-------|--------|----------------|--------|\n| `SafeResolve` | `os.Lstat`, `filepath.EvalSymlinks` (metadata only) | None | None | Walks ancestors upward until one exists; no data read/written. |\n| `AcquireLock` | None | `O_CREATE\\|O_EXCL` + PID bytes via `fmt.Sprintf(\"%d\\n\", os.Getpid())` | None | Atomic creation on success; cleans up file handle and removes partial state if PID write fails before returning error. |\n| `SimpleLock.Unlock()` | None | `l.file.Close()` (flushes buffer) | `os.Remove(l.lockPath)` β€” best-effort, non-blocking | Idempotent. Swallowed close errors; remove failures are discarded unless they occur after a successful close and removal itself fails. |\n| `EnsureGitignoreHasLockfile` | `os.ReadFile(gitignorePath)` | Append-mode open + write string | None | Creates `.gitignore` if absent (append semantics). Appends only when lock filename is not already present. |\n\n### Network / External Process\n\nNone. The package performs no HTTP, gRPC, socket, or IPC operations. No external process interaction beyond reading/writing files within the repository boundary.\n\n---\n\n## Error Handling Patterns\n\n### Sentinel Errors (Unwrapped)\n\n| Condition | Returned Value | Rationale |\n|-----------|---------------|-----------|\n| Lock held by another process | `\"lock at %s is already held by another process...\"` | Actionable message; caller needs exact match or substring search. |\n| Path traversal detected | `\"security violation: path traversal detected: %q\"` | Security-critical; never wrapped for unambiguous detection. |\n\n### Wrapped Errors (Standard Library)\n\n| Condition | Wrapping Strategy | Notes |\n|-----------|------------------|--------|\n| `filepath.Abs` fails | `%w` | Root is not absoluteβ€”caller should handle gracefully. |\n| `filepath.EvalSymlinks` on root fails | `%w` | Symlink resolution failure for the repository boundary itself. |\n| `os.Lstat` returns non-nil, non-IsNotExist error | `%w` | Unexpected stat error on an ancestor component. |\n| `EvalSymlinks` on ancestor fails | `%w` | Ancestor symlink resolution failure during upward walk. |\n| `.gitignore` read (not IsNotExist) | `%w` | Read of existing gitignore encounters unexpected I/O error. |\n| Append/open for `.gitignore` fails | `%w` | File open or write failure for the ignore file. |\n\n### Swallowed Errors (Best-Effort Cleanup)\n\nIf `l.file.Close()` inside `Unlock()` returns an error, it is replaced with `nil` and cleanup proceeds unconditionally. If `os.Remove(l.lockPath)` fails after a successful close, the remove-error is discarded unless both close-succeeded-and-remove-failedβ€”but this narrow path effectively swallows non-`IsNotExist` errors in practice. The lockfile removal happens regardless; any failure that isn't `IsNotExist` is preserved in a local variable but only surfaces when close succeeded and remove failedβ€”which is treated as swallowed behavior: best-effort cleanup.\n\n### Lock Acquisition Failure Cleanup\n\nIn `AcquireLock`, if writing the PID to the newly created file fails, the function calls `f.Close()` then `os.Remove(lockPath)` before returning an error. The file handle and lockfile are cleaned up so no stale partial state remains on disk; the returned error is wrapped with `%w` for context.", - "internal/tools": "# `internal/tools` β€” File System \u0026 Git Operations Module\n\n## Module Responsibility\n\nThis module provides two complementary interfaces for repository-aware operations: safe file I/O with TOCTOU race mitigation, and Git process abstraction for executing commands within the working tree. Both modules operate on a single root path (`repoRoot`) without shared mutable state, and neither uses concurrency primitives β€” all functions are goroutine-safe by construction.\n\n---\n\n## `file_tools.go` β€” Safe File I/O \u0026 Discovery\n\n### Core Functions\n\n| Function | Input | Output |\n|---|---|---|\n| [`ReadFileSafely`](file:///path/to/file_tools.go#L15-L64) | `repoRoot`, `virtualPath` `string` | `[]byte`, `error` |\n| [`WriteFileSafely`](file:///path/to/file_tools.go#L70-L132) | `repoRoot`, `virtualPath` `string`; `content` `[]byte` | `error` |\n| [`LoadGitignore`](file:///path/to/file_tools.go#L138-L164) | `repoRoot` `string` | `[]string`, `error` |\n| [`ShouldIgnoreFile`](file:///path/to/file_tools.go#L170-L236) | `repoRoot`, `relPath` `string`; `gitIgnore` `*ignore.GitIgnore`; `ignoredExtensions` `[]string` | `bool` |\n| [`DiscoverCodeFiles`](file:///path/to/file_tools.go#L242-L291) | `repoRoot` `string`; `ignores`, `ignoredExtensions` `[]string` | `[]string`, `error` |\n| [`IsBinaryFile`](file:///path/to/file_tools.go#L297-L315) | `path` `string` | `bool` |\n\n### TOCTOU Race Mitigation Pattern\n\nBoth `ReadFileSafely` and `WriteFileSafely` implement a Time-Of-Check-Time-Of-Use safety pattern:\n\n1. Resolve the virtual path to an absolute safe path via `SafeResolve`\n2. Open the file descriptor first\n3. Perform `lstat` on the resolved path (no symlink following)\n4. Perform `fstat` on the open file descriptor (follows symlinks)\n5. Verify that `lstat` and `fstat` reference the same inode β€” mismatch indicates a symlink race between opening and checking\n\nBoth operations reject any symlink detection as a security violation. On success, `ReadFileSafely` reads all content via buffered I/O; on success, `WriteFileSafely` truncates to zero first (safe only because symlinks were ruled out), then writes the provided content.\n\n### Error Handling Contract\n\n| Function | Wrapped Errors (`%w`) | Swallowed | Sentinel Unwrapped |\n|---|---|---|---|\n| `ReadFileSafely` | Yes, for every I/O error | No | Symlink race check |\n| `WriteFileSafely` | Yes, for every I/O error | No | Symlink race check |\n| `LoadGitignore` | No | Missing file β†’ `(nil, nil)` | None |\n| `ShouldIgnoreFile` | No | `SafeResolve` err β†’ returns `true` | None |\n| `DiscoverCodeFiles` | No | Extensive β€” walk callback errors swallowed at each node | None |\n| `IsBinaryFile` | No | All open/read errors β†’ `false` | None |\n\n### File Discovery Algorithm (Multi-Layer Ignore Rules)\n\nA file is considered \"ignored\" if it matches **any** of these conditions:\n\n- Matches a compiled gitignore pattern list\n- Has a path component starting with `.` or ending in `.egg-info`\n- Has an extension matching the user-provided ignored extensions list (case-insensitive, supports both `.` prefix and suffix forms)\n- Is detected as binary by reading up to 1024 bytes and scanning for null byte (`\\x00`)\n\nIf any single check returns true, the file is excluded. This is a **union** of ignore conditions.\n\nThe primary business rule is to recursively walk the repository root and discover high-signal source code files while filtering out noise: build artifacts, dependency directories, output files, dot-prefixed hidden paths, `.egg-info` markers, binary files, and any user-defined ignore patterns (via gitignore compilation).\n\n### Binary Detection Rule\n\nA file is considered binary if its first 1024 bytes contain a null byte (`\\x00`). Files that cannot be opened are treated as non-binary (returns false), avoiding the loading of entire files into memory for classification.\n\n---\n\n## `git_tools.go` β€” Git Process Abstraction\n\n### Core Functions\n\n| Function | Input | Output |\n|---|---|---|\n| [`RunGit`](file:///path/to/git_tools.go#L15-L64) | `repoRoot string`, variadic `args ...string` | `(string, error)` |\n| [`VerifyGitRepo`](file:///path/to/git_tools.go#L70-L89) | `repoRoot string` | `error` |\n\n### Execution Model\n\nBoth functions spawn the `git` binary via `exec.Command(\"git\", ...)` with `--no-pager`, restricting execution to the provided root directory. Both capture stdout and stderr into buffers synchronously. No standard library packages beyond `bytes`, `fmt`, `os/exec`, `strings` are imported.\n\n### Error Handling Contract\n\n| Function | Failure Path | Behavior |\n|---|---|---|\n| `RunGit` | Exit code β‰  0 | Wraps error: `git command failed: %v, stderr: %s`. Returns `(string, error)` with stdout and wrapped error. No panic. |\n| `RunGit` | Exit code 0 | Returns `(trimmedOut, nil)`. Stdout is whitespace-trimmed; stderr is discarded on success. |\n| `VerifyGitRepo` | `RunGit(...)` returns error | Wraps it: `not a git repository (or any of the parent directories)`. The original stderr content is lost in this wrapper. Returns only the wrapped sentinel-like error. No panic. |\n\n### Repository Integrity Check\n\nA directory is considered a valid Git working tree only if `git rev-parse --is-inside-work-tree` succeeds; any failure indicates the path is outside a Git working tree or not a git repository at all. The verification phase delegates to the execution phase with specific arguments, interpreting success as confirmation of repository validity and failure as an assertion that the directory is not a Git repository.\n\n---\n\n## Data Flow Summary\n\n```\nDiscoverCodeFiles(repoRoot, ignores, ignoredExtensions)\nβ”œβ”€β”€ Compile gitignore from ignore lines β†’ gitIgnore object (in-memory, no I/O)\n└── filepath.WalkDir(repoRoot):\n β”œβ”€β”€ Skip repo root itself (first entry)\n β”œβ”€β”€ If directory:\n β”‚ β”œβ”€β”€ Check name against dot-prefix / .egg-info / gitIgnore\n β”‚ └── If any match β†’ Skip entire subtree\n └── If file:\n └── ShouldIgnoreFile check:\n β”œβ”€β”€ gitignore matches? β†’ ignore\n β”œβ”€β”€ path component starts with \".\" or ends \".egg-info\"? β†’ ignore\n β”œβ”€β”€ extension in ignoredExtensions list? β†’ ignore\n β”œβ”€β”€ binary? β†’ ignore\n └── otherwise β†’ add to result list\n\nResult: flat list of relative paths representing source code files that pass all safety and filtering rules.\n```" - } -} \ No newline at end of file diff --git a/wiki/architecture.md b/wiki/architecture.md deleted file mode 100644 index 68a0944..0000000 --- a/wiki/architecture.md +++ /dev/null @@ -1,187 +0,0 @@ -# Architecture Overview β€” Code-Reducer - -## System Boundaries & Module Interaction Map - -``` -β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ cmd (CLI) β”‚ β”‚ internal/engine β”‚ β”‚ internal/config β”‚ -β”‚ β”‚ β”‚ (Pipeline Engine) β”‚ β”‚ (Config Resolution) β”‚ -β”‚ β€’ root.go β”‚ β”‚ β”‚ β”‚ β”‚ -β”‚ β€’ setup.go │────▢│ β€’ runner.go │◀───│ β€’ ResolveConfig β”‚ -β”‚ β€’ init.go β”‚ β”‚ β€’ orchestrator β”‚ β”‚ β€’ LoadConfig β”‚ -β”‚ β€’ update.go β”‚ β”‚ β€’ tree.go β”‚ β”‚ β€’ SaveConfig β”‚ -β”‚ β”‚ β”‚ β€’ chunking.go β”‚ β”‚ β€’ ExtractionSteps β”‚ -β”‚ β”‚ β”‚ β€’ synthesize.go β”‚ β”‚ β€’ Config β”‚ -β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ - β”‚ - β–Ό - β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” - β”‚ LLM Endpoint β”‚ - β”‚ (Ollama-compat) β”‚ - β”‚ /api/chat β”‚ - β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ -``` - -**Execution flow:** `cmd.RootCmd.Execute()` β†’ config resolution (`internal/config.ResolveConfig`) β†’ engine dispatch (`engine.Runner.Run` with `"init"` or `"update"`) β†’ LLM calls through `LLMClient`. All three packages are decoupled at boundaries: the CLI package wires subcommands and suppresses cobra defaults; the config package resolves values through a four-tier precedence pipeline; the engine package owns the map-reduce pipeline, tree construction, chunking, and synthesis. - ---- - -## Module Responsibility & Data Flow - -### `cmd` β€” Entry Point & Subcommand Wiring - -The CLI layer registers three subcommands (`init`, `update`, `setup`) via package-level `init()` calls. Two persistent flags are bound globally: `--model-id` (via `PersistentFlags` + `modelIdFlag`) and `--num-ctx`. The root command performs environment validation, config resolution, implicit setup on first run when stdin is a TTY, mode-gated lifecycle checks delegated to the runner/engine layer, signal handling via `signal.NotifyContext`, and dispatches to `engine.NewRunner.Run()`. - -All failure paths return errors propagated through cobra. Cobra's default error printing is suppressed (`SilenceErrors: true`, `SilenceUsage: true`), so callers handle errors explicitly. No panic calls are observed in this package. - -**Init subcommand** registers the entry point and delegates to `executeCommand("init")`, which performs repository scan and wiki page generation outside this package boundary. All substantive logic is deferred externally. - -**Update subcommand** similarly exports no public functions; it registers via `init()` and delegates to `executeCommand("update")`. The actual update logic lives outside this file. - -**Setup flow** (`RunSetupFlow(repoRoot)`) generates `.code-reducer.yaml` through an interactive prompt loop: resolve existing state, prompt iteratively for each setting (model ID β†’ string, base URL β†’ string, context size β†’ integer parsed via `strconv.Atoi`, custom directories/files to ignore β†’ comma-separated list, file extensions to ignore β†’ comma-separated list, documentation directory β†’ string), preserve existing values on empty input, support reset semantics where `"clear"` or `"none"` wipes the list entirely, swallow read failures silently (logged to stderr, continue with existing values), and persist final config via `config.SaveConfig`. - -### `internal/config` β€” Configuration Resolution - -Implements a four-tier precedence pipeline. Each layer overwrites prior values when present; missing layers are transparently skipped. Three properties resolve independently: `ModelID`, `OllamaBaseURL`, `OllamaNumCtx`. The package also structures the analysis pipeline into named **extraction steps** (`ExtractionStep`), each carrying a distinct prompt template. All non-structural state (resolved config, local variables) is ephemeral β€” scoped to function calls and returned as values. No package-level mutable state survives beyond initialization. Concurrency primitives are absent; the package is single-threaded in design. - -#### Configuration Loading (`LoadConfig`) - -Reads YAML at the resolved path. File-read failure β†’ pass-through error. YAML parse failure β†’ wrapped as `"failed to parse yaml config: %w"`. Returns non-nil `*Config` on success, initialized with zero values before parsing so callers receive a populated object even for empty mappings. - -#### Configuration Resolution (`ResolveConfig`) - -Top-level entry point: (1) calls `LoadConfig`, swallows non-existent-file sentinel (`os.IsNotExist(err)`), initializes empty `Config{}`, other errors propagate wrapped as `"failed to load configuration file:"`; (2) layers defaults (hard-coded constants); (3) layers environment variable overrides using `os.Getenv` for `CODE_REDUCER_MODEL_ID`, `OLLAMA_BASE_URL`, `OLLAMA_NUM_CTX` β€” non-empty values used, empty strings fall through; env read errors swallowed; (4) applies CLI flags (`modelIdFlag`, `numCtxFlag`) β€” flag value parsed via `strconv.Atoi`; only `err == nil && n > 0` accepted, invalid or non-positive retain prior layer's value silently. - -#### Configuration Persistence (`SaveConfig`) - -Serializes config struct back to disk using `os.WriteFile` with `0600` permissions. Write success β†’ `nil`. Parse failure (marshal YAML) β†’ wrapped as `"failed to marshal yaml: %w"`. File-write failure β†’ wrapped as `"failed to write config file: %w"`. Writes directly, no temp-file + rename; partial writes can leave a truncated file on disk. - -#### Defaults & Constants - -| Constant | Value | Purpose | -|---|---|---| -| `CodeReducerModelIdEnvKey` | `"CODE_REDUCER_MODEL_ID"` | Overrides `Config.ModelID` when set | -| `OllamaBaseUrlEnvKey` | `"OLLAMA_BASE_URL"` | Overrides `Config.OllamaBaseURL` when set | -| `OllamaNumCtxEnvKey` | `"OLLAMA_NUM_CTX"` | Overrides `Config.OllamaNumCtx`; must parse as positive integer | -| `OllamaDefaultBaseURL` | `"http://localhost:11434"` | Fallback base URL | -| `OllamaDefaultModelID` | `"ornith:9b"` | Default model identifier | -| `OllamaDefaultNumCtx` | `8192` | Default context size | -| `ConfigFileName` | `.code-reducer.yaml` | Filename relative to CWD | - -Three slices declared at package init and never mutated: `DefaultIgnores`, `DefaultIgnoredExtensions`, `DefaultExtractionSteps`. - -#### Path Resolution & Existence Check - -**`getConfigPath(cwd string) (string, error)`** β€” constructs full path to `.code-reducer.yaml` by joining `cwd` with `ConfigFileName`. Returns non-nil error only on I/O failure during construction; normal cases return `nil`. - -**`ConfigExists(cwd string) bool`** β€” returns whether config file exists at resolved path. Internally calls `os.Stat`; all errors (permission denied, I/O failures, etc.) are swallowed and treated as "does not exist". Returns no error. - -### `internal/engine` β€” Code Documentation Pipeline Engine - -Implements a Map-Reduce pipeline that generates and maintains technical documentation for a Go codebase using Ollama-compatible LLM inference endpoints. The engine produces three artifacts per run: `architecture.md` (global overview), `quickstart.md` (onboarding guide), and per-module API summaries under `modules/`. Execution is gated by a repository-level lock to prevent concurrent runs against the same root; state persistence across invocations is handled by an on-disk metadata cache. - -#### Pipeline Orchestration β€” Runner Entry Point (`runner.go`) - -The public entry point for all pipeline invocations: repository isolation, concurrency guard acquisition via `security.AcquireLock(repoRoot)`, LLM client construction from config, and mode-gated dispatch to the appropriate pipeline method. - -``` -Run(ctx, repoRoot, mode string, onEvent func(Event)) error - β”œβ”€ ensure .gitignore lockfile entry (best-effort; failure logged as warning) - β”œβ”€ acquire repository-level lock via security.AcquireLock(repoRoot) - β”‚ └─ defer lock.Unlock() β€” released on any return path including errors - β”œβ”€ build *LLMClient from r.cfg.ModelID, BaseURL, NumCtx - β”œβ”€ switch mode: - β”‚ case "init": client.RunInit(ctx, repoRoot, r.cfg, onEvent) - β”‚ case "update": client.RunUpdate(ctx, repoRoot, r.cfg, onEvent) - β”‚ default: return fmt.Errorf("unsupported mode: %s", mode) -``` - -The `cfg *config.Config` field is read-only within this file; all mutations to the runner's configuration originate before construction. The lock returned by `security.AcquireLock(repoRoot)` is a local variable with deferred cleanup β€” its internal type and mutual exclusion semantics are not observable from this source alone. - -#### Orchestrator Pipelines (`orchestrator.go`) - -**Full Run (`RunInit`)** executes an all-or-nothing regeneration regardless of cache state: (1) setup β€” load `.gitignore` via `tools.LoadGitignore(repoRoot)`; merge with user ignore list; load metadata cache from disk via `cache.loadMetadataCache()`. Failures logged as warnings, not returned. (2) map phase β€” discover code files via `tools.DiscoverCodeFiles(...)`, compute SHA-256 per file via `computeSHA256(repoRoot, f)`, build directory tree via `buildTree(codeFiles)`. Hash computation errors silently skipped in loops; only stored if hash computed successfully. (3) mark all affected β€” recursively flag every node as affected (init starts fresh). (4) reduce phase β€” call `synthesizeNode` bottom-up on the tree to produce a root summary string and per-module summaries via LLM calls through `c.CallLLM`. (5) generate standard docs β€” produce `architecture.md` and `quickstart.md` from the root summary via two additional `CallLLM` invocations: one for architecture, one for quickstart. Writes via `tools.WriteFileSafely`; errors wrapped with context (e.g., `"failed to write quickstart.md: %w"`). (6) update AGENTS.md β€” read existing file via `tools.ReadFileSafely(repoRoot, "AGENTS.md")`. If missing, create with guidelines header. If present but does not contain "AI Agent Guidelines", append separator + new content block. Only writes when content actually differs. Errors: if read fails but write succeeds β†’ `"failed to write AGENTS.md: %w"`; if both fail β†’ wrapped errors chain. (7) teardown β€” persist metadata cache via `saveMetadataCache(repoRoot, docsDir, cache)`. Failure logged as warning, function always returns nil at end regardless of this failure. - -**Incremental Run (`RunUpdate`)** executes only re-generations for directories whose contents actually changed: (1) setup β€” load pipeline state from metadata cache (same path as init). (2) map phase β€” discover code files, hash each via `computeSHA256`, build tree. (3) change detection β€” compare current file hashes against cached hashes in `cache.Files`: new file absent from cache β†’ `FileChange{Status: "Added"}`; existing file with different hash β†’ `FileChange{Status: "Modified"}`; cached file no longer in codebase β†’ `FileChange{Status: "Deleted"`} (also removed from cache). (4) prune stale modules β€” remove any module summaries whose directory is no longer active, delete their markdown files on disk via `os.Remove(absModuleFile)` (error assigned to `_`, never logged). (5) propagate affected β€” run tree-aware propagation (`propagateAffected`/`determineAffected`) to mark directories affected by changes. Errors from these calls are discarded without capture. (6) early exit β€” if zero directories are affected, pipeline ends with "up to date" status, no further synthesis occurs. (7) conditional reduce β€” if root-level change detected or global docs don't exist on disk (`os.Stat(absArch)`/`os.Stat(absQs)`), run full synthesis; otherwise skip and reuse existing `architecture.md`/`quickstart.md`. (8) teardown β€” save updated cache via `saveMetadataCache(repoRoot, docsDir, cache)`. - -#### Directory Tree Construction & Affected Propagation (`tree.go`) - -**Type Definitions:** - -| Type | Fields | Purpose | -|------|--------|---------| -| `FileChange` | `Path string`, `Status string` | Models a file operation with path and status (`Added`, `Modified`, `Deleted`). Consumed by `determineAffected`. | -| `DirNode` | `Path string`, `Files []string`, `Children map[string]*DirNode` | Hierarchical tree node holding files at that level and child directories in its subtree. Produced by `buildTree`. | - -**Functions (all unexported):** - -- **`buildTree(codeFiles)`** β€” converts a flat list of file paths into nested `*DirNode` structure, preserving directory nesting by splitting on `/`. Appends to `DirNode.Files`; populates `DirNode.Children` via map assignment. -- **`propagateAffected(tree, changeset)`** β€” traverses the tree recursively: if any direct child of the current node is in the changeset, mark it affected. Accumulates affected status upward through ancestors. Returns mutated `affectedDirs`. -- **`determineAffected(tree, changeset)`** β€” computes module path for each directory via `ToSafeMarkdownFilename`, checks existence on disk via `os.Stat(absModulePath)`, reads metadata cache (`cache.Modules[n.Path] == ""`), marks node affected if any check fails. Only `os.IsNotExist` is acted on; all other errors are silently dropped. - -#### LLM Client Abstraction & Response Parsing - -**Type: `LLMClient`** - -```go -type LLMMClient struct { - ModelID string - BaseURL string - NumCtx int - HTTPClient *http.Client // default transport, no custom config (no proxy/redirect/TLS) -} -``` - -Constructed via `NewLLMClient(modelID, baseURL, numCtx)` with a 10-minute HTTP timeout. Fields set once; never reassigned within any method. - -**Methods on `*LLMClient`:** - -- **`CallLLM(ctx context.Context, systemPrompt string, messages []Message, jsonFormat bool) (string, error)`** β€” full lifecycle: prepend system prompt as first message with role `"system"` β†’ serialize into Ollama `/api/chat` JSON payload with model ID and context window size β†’ non-streaming HTTP POST β†’ response body fully read on 200 OK or truncated at 1 KB via `io.LimitReader(resp.Body, 1024)` on non-OK status. Returns only assistant content string; returns empty string on failure. - -**Error behavior:** - -| Failure path | Error type | Wrapping strategy | -|---|---|---| -| JSON marshal fails in request prep | `fmt.Errorf("failed to marshal request: %w", err)` | Wrapped with `%w` | -| `http.NewRequestWithContext` fails | raw error from `NewRequestWithContext` | **Not wrapped** β€” passed through as-is | -| HTTP Do (network-level failure) | raw error from `c.HTTPClient.Do(req)` | **Not wrapped** | -| Body read fails on 200 OK path | raw error from `io.ReadAll` | **Not wrapped** | -| JSON unmarshal fails on response | `fmt.Errorf("failed to parse response: %w", err)` | Wrapped with `%w` | -| Non-OK status (body truncated at 1 KB) | `fmt.Errorf("ollama api error: status %d, response: %s", ...)` | New error constructed; body discarded if read fails | - -- **`GetBaseSystemPrompt() string`** β€” returns the fixed base role definition and defensive guidelines inherited by every LLM call regardless of task type. -- **`GetDefaultSystemPrompt(command string) string`** β€” appends command-specific instructions based on the command string: two named variants exist (`module_synthesis`, `architecture`); any other value receives only the base layer. - -#### Response Parsing Layer (internal, unexported helper `extractBalancedJSON`) - -1. Markdown code fences (` ``` `) surrounding JSON must be stripped before parsing β€” handled by `StripOuterMarkdownFence`. -2. Balanced bracket matching required β€” strings containing `{`, `[`, `}`, `]` literals (including escaped characters like `\n`) must not break extraction logic β€” handled by internal rune-level scanner with escape state tracking inside string literals. -3. When multiple balanced JSON candidates exist in a single response, each is attempted individually until one succeeds, or the entire trimmed string is tried as a final fallback β€” handled by `CleanJSONResponse`/`UnmarshalJSONResponse`. - -**Functions:** - -| Function | Behavior | Error surface | Swallowed? | -|---|---------|---------------|------------| -| `StripOuterMarkdownFence(s string) string` | Strips surrounding markdown fences; falls back to trimmed input if none match | None β€” returns string only | N/A (no error return) | -| `CleanJSONResponse(s string) string` | Iterates stripped content, calls `extractBalancedJSON`, returns first successful candidate or original trimmed input unchanged | None β€” returns string only | Yes β€” unbalanced JSON inside response is swallowed silently | -| `UnmarshalJSONResponse(s string, target interface{}) error` | Multi-layered attempt: (1) collect balanced candidates, try each into target; keep last error in `lastErr`; (2) if any succeeded, return nil; otherwise try entire trimmed string; (3) terminal errors β€” `"failed to unmarshal JSON candidates: %w"` or `"no valid JSON found in response"` | Explicit (`error`) | Partial parse failures collected and re-emitted via `%w` on first path; "no valid JSON" covers second path | - -#### Synthesis Engine & Chunking (internal, all unexported) - -The system processes extracted code/fact items in batches constrained by LLM context limits (`NumCtx * 3`). Three distinct processing modes exist: - -**Module Synthesis (`reduceInChunks`)** β€” synthesizes architectural-level nodes into architecture descriptions using a `"module_synthesis"` system prompt. Each batch represents evidence contributing to understanding the module's design intent. Response passes through `StripOuterMarkdownFence`. Errors wrapped with `"LLM error during synthesis: %w"`. - -**File Fact Consolidation (`reduceFileFacts`)** β€” consolidates extracted facts about a specific step within a file with deduplication and merging. System prompt built locally by concatenating `c.GetBaseSystemPrompt()` + fixed role string. Response passes through `StripOuterMarkdownFence`. Errors wrapped with `"LLM error during file fact consolidation: %w"`. - -**Recursive Convergence (`reduceItems`)** β€” core business rule: reduce items in batches β†’ collect intermediate results β†’ recursively apply reduction until everything converges into a single output string. Multi-pass approach ensures information is progressively synthesized rather than lost across multiple LLM calls. Termination guaranteed when all data fits within one batch (short-circuits at top with `len(batches) == 1`). Recursion terminates because: (1) single-batch case short-circuits at top (`len(batches) == 1`); (2) recursive step processes each batch through itself again with same `maxChars`. - -**Graceful Truncation Policy:** when content exceeds the context window, single oversized items get truncated with `\n...[truncated]` marker appended; when multiple items exceed limits collectively, each gets capped at `maxChars / len(items)` characters before being processed as one batch. Infinite-loop guard: if batch size equals item count for all items (can't merge), falls back to per-item truncation using `allowedPerItem = maxChars / len(items)`. Still silent β€” no error returned. - -**Overlapping Chunk Support (`chunkTextWithOverlap`)** β€” utility for splitting raw text into overlapping chunks to maintain context continuity between adjacent segments β€” useful when feeding source material directly into LLMs without pre-processing through the chunking logic. No I/O. Silent swallow: `maxRunes <= 0` β†’ returns full text as single chunk; `overlapRunes >= maxRunes` β†’ clamps to half without warning; final partial chunk reaching end of runes β†’ breaks loop with no error or truncation marker (unlike `reduceItems`). - -#### Hierarchical Module Synthesis (`synthesizeNode`, all unexported) - -Performs recursive synthesis of directory structures into hierarchical summaries β€” transforming raw source code and subdirectory metadata into consolidated documentation at each level of a module tree. - -**Algorithmic Flow:** (1) context check β†’ cache reuse gate: if current node not in `affectedDirs` AND cached module summary exists, return immediately without processing. Short-circuits unaffected directories. (2) recursive child synthesis (bottom-up): sort children alphabetically, recursively call `synthesizeNode` on each child directory; results collected into map keyed by child name. (3) file processing loop: for each file in current node β€” hash-first cache lookup (if precalculated hash exists and matches cached SHA-256 β†’ reuse cached facts without reading disk); read + hash fallback otherwise (read via `tools.ReadFileSafely`, compute SHA-256, update cache entry if already existed with that hash); extraction pipeline on cache miss (chunk content using dynamic limit derived from `c.NumCtx Γ— 4`...). \ No newline at end of file diff --git a/wiki/modules/cmd.md b/wiki/modules/cmd.md index a797f48..1fcbb77 100644 --- a/wiki/modules/cmd.md +++ b/wiki/modules/cmd.md @@ -1,217 +1,144 @@ -# Architecture: `cmd` Package β€” CLI Command Wiring & Lifecycle Management +# Module: `cmd` β€” CLI Command Registration, Lifecycle Management, and Interactive Configuration -The `cmd` package implements the top-level cobra command hierarchy for Code-Reducer, a documentation-generation agent that writes and maintains project wikis by calling an LLM to produce markdown docs from source code. This module is responsible for registration of subcommands (`init`, `setup`, `update`), resolution of persistent configuration flags, environment validation (Git repo check), signal handling, and delegation to the engine's runner via a shared `executeCommand` function referenced but not defined within this package boundary. +## Responsibility -## Data Flow +The `cmd` package implements the command-layer for the **Code-Reducer** documentation agent. It registers Cobra subcommands (`code-reducer init`, `code-reducer update`, `code-reducer setup`) and delegates execution of each to a shared `executeCommand` handler whose implementation resides outside this package boundary. The package also owns: -``` -User CLI invocation - β†’ Cobra parses subcommand - β†’ cmd/init.go : executeCommand("init") β†’ repository scan + wiki scaffolding - β†’ cmd/setup.go : RunSetupFlow(repoRoot) β†’ interactive config prompts β†’ disk write - β†’ cmd/update.go : executeCommand("update") β†’ incremental wiki regeneration -``` +- Root command construction with environment validation, signal handling, and event-driven output streaming. +- A lifecycle state machine that enforces ordering constraints between subcommands (init before update; setup as prerequisite for run). +- Interactive configuration prompts for first-time or reconfiguration scenarios. -All three subcommand handlers route execution through a single function reference (`executeCommand`) whose implementation is not present in this package. No business logic, validation, or data processing occurs within the registration files themselves; they serve only as wiring between cobra's command framework and the out-of-scope executor. +No exported types, variables, or functions are defined in this package. All identifiers have lowercase first letters and remain internal to `cmd`. --- -## Root Command Initialization & Persistent Flags +## Command Registration Layer (`cmd/init.go`, `cmd/update.go`) -### `RootCmd` (exported) +Both files exist solely as command wiring stubs. Each declares a local `*cobra.Command` variable (unexported) with its `RunE` closure delegating to an external `executeCommand` function: -- Initialized with `Use="code-reducer"`, short/long descriptions, and `CompletionOptions{DisableDefaultCmd: true}`. -- Two persistent flags registered via `PersistentFlags`: - - `--model-id` β€” bound to package-level variable `modelIdFlag` - - `--num-ctx` β€” bound to package-level variable `numCtxFlag` - -### Initialization Sequence in `root.go` - -1. **Environment validation** β€” `os.Getwd()` resolves the current working directory; failure is wrapped with `%w` and returned immediately: `fmt.Errorf("failed to get current working directory: %w", err)`. -2. **Configuration resolution** β€” merges three sources: CLI flags (`--model-id`, `--num-ctx`), environment defaults, and an optional persistent config file (loaded via `internal/config`). The resolved config drives all subsequent behavior; details of this merge are delegated outside the package boundary. -3. **Implicit setup on first run** β€” if the persistent config file does not exist yet *and* stdin is a terminal (`isatty.IsTerminal(os.Stdin.Fd())` / `isatty.IsCygwinTerminal(os.Stdin.Fd())`), an automatic initial configuration step triggers without requiring user intervention. On non-TTY (CI / piping), this step is skipped and the caller must configure manually via `setup`. -4. **Mode-gated lifecycle checks** β€” enforced only within the runner/engine layer; at registration time, no validation of project state occurs here. The two modes (`init`, `update`) are mutually exclusive on an already-initialized project; enforcement lives in the external executor. -5. **Graceful shutdown** β€” installs SIGINT / SIGTERM handlers via `signal.NotifyContext`. A deferred `stop()` cancels the context; any in-flight work that respects the cancelled context will exit. No error-return path exists for signal handling within this file. -6. **Execution loop** β€” delegates to `engine.NewRunner.Run()` with the resolved config and current directory. The runner emits typed events (`status`, `error`, default). - -### Side Effects & Error Handling in `root.go` - -| Operation | Direction | Mechanism | Notes | -|---|---|---|---| -| `os.Getwd()` | Read OS state | Filesystem call | Wrapped with `%w`; caller handles failure. | -| `isatty.IsTerminal(os.Stdin.Fd())` / `isatty.IsCygwinTerminal(os.Stdin.Fd())` | Read OS state | Stdio file descriptor inspection via third-party library (`go-isatty`) | Used only for the guard check before setup flow; result is not persisted. | -| `fmt.Println(ev.Message)` (2 occurrences) | Write to stdout | Standard library `fmt` package | Triggered inside the callback passed to `runner.Run()` when event type is `"status"` or default case. | -| `fmt.Fprintf(os.Stderr, "Error: %s\n", ev.Message)` | Write to stderr | Standard library `fmt` package | Triggered inside the callback when event type is `"error"`. | - -**Cobra output suppression:** `RootCmd` has `SilenceErrors: true` and `SilenceUsage: true`. Errors that would otherwise be printed by cobra's default handler are suppressed β€” callers must handle errors explicitly (which this function does). No panic calls observed; all failure paths return errors propagated to cobra. - -### Mutable State & Concurrency in `root.go` - -| Variable | Scope | Mutability | Notes | -|---|---|---|---| -| `modelIdFlag` | Package-level (`var`) | Mutated once via `StringVar` in `init()` | Set by flag binding; value consumed in `executeCommand()` without further writes. | -| `numCtxFlag` | Package-level (`var`) | Mutated once via `StringVar` in `init()` | Same pattern as above. | +| Identifier | Kind | Behavior | +|---|---|---| +| `initCmd` (`cmd/init.go`) | `*cobra.Command` | Registered under `RootCmd`; `RunE` invokes `executeCommand("init")`. No state mutation after declaration. | +| `updateCmd` (`cmd/update.go`) | `*cobra.Command` | Registered under `RootCmd`; `RunE` invokes `executeCommand("update")`. No state mutation after declaration. | -No other fields or structs are mutated within this file after their initial assignment. No mutexes, channels, atomic operations, or other concurrency primitives appear here; the signal notification context (`signal.NotifyContext`) is used for cancellation but does not protect shared mutable stateβ€”each `executeCommand` invocation runs sequentially and consumes its own local variables. +The domain concept for the init flow is **project initialization** β€” scanning a repository to generate initial wiki documentation (markdown pages). The update flow targets files changed since the last documented commit and updates corresponding wiki pages. Neither file contains business logic; both pass execution to `executeCommand` whose error handling strategy cannot be determined from this source alone because the function is out of scope here. --- -## Interactive Configuration Setup Flow (`setup.go`) +## Root Command & Lifecycle State Machine (`cmd/root.go`) -### Exported Functions +### Core Domain Concepts -| Function | Signature | Notes | -|---|---|---| -| `RunSetupFlow` | `(repoRoot string) error` | Interactive setup flow that generates a `.code-reducer.yaml` configuration file. | -| `promptStringList` | `(reader *bufio.Reader, promptMsg string, existingList []string) []string` | Reads comma-separated list input from stdin. Returns existing values on I/O error or empty/"clear"/"none" input. | +The package-level variable `RootCmd` is a `*cobra.Command` initialized with: -### Internal (Unexported) Functions +- `Use` = `"code-reducer"` +- `Short`, `Long` descriptions (contents not exposed here) +- `DisableDefaultHelpCommand` +- `CompletionOptions{DisableDefaultCmd: true}` +- `SilenceUsage`, `SilenceErrors` both set to `true` -| Function | Signature | Notes | -|---|---|---| -| `init` | `()` | Registers `setupCmd` with `RootCmd`. | -| `promptString` | `(reader *bufio.Reader, promptMsg string, existingVal string) string` | Reads single-line text input from stdin. Returns existing value on I/O error or empty input. | +The root command owns four behavioral responsibilities beyond subcommand registration: -### Internal State +1. **Environment validation** β€” verifies the current working directory is a valid Git repository via an external `tools.VerifyGitRepo(repoRoot)` call; failure is hard (no soft fail). +2. **Configuration resolution** β€” merges three sources in order of precedence: persisted config file, CLI flag overrides (`--model-id`, `--num-ctx`), then defaults. The merged result drives all downstream behavior. +3. **Lifecycle state machine** β€” enforces ordering constraints between subcommands (init requires project NOT yet initialized; update requires project IS already initialized). Invalid transitions fail with explicit sentinel messages. +4. **Graceful termination** β€” installs signal handlers for `SIGINT` and `syscall.SIGTERM`. A context is created via `signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)` whose returned `ctx` and `stop` function manage lifecycle; deferring `stop()` cancels the context on return. +5. **Event-driven execution** β€” spawns an engine runner that emits events (status, error, etc.) which are streamed to stdout/stderr in real time rather than collected after completion. The callback passed to `runner.Run` is a side-effect only: it writes events but does not propagate them back as errors. Error state comes from `err != nil` at the end of the run path. -| Identifier | Type | Notes | -|---|------|---| -| `setupCmd` | `\*cobra.Command` | The `setup` sub-command registered with the root cobra command. | +### Mutable State (`cmd/root.go`) -### High-Level Algorithmic Flow in `RunSetupFlow` +| Variable | Type | Mutated? | Notes | +|---|---|---|---| +| `modelIDFlag` (string) | package-level | Once, during flag registration in `init()` via `StringVar` | Not mutated after declaration. | +| `numCtxFlag` (string) | package-level | Once, during flag registration in `init()` via `StringVar` | Same pattern as above. | -1. **Resolve existing state** β€” attempt to load any previously saved config from the current working directory via `config.LoadConfig(repoRoot)`. If none exists, all prompts default to built-in constants (Ollama defaults for model ID, base URL, context size; `"wiki"` for docs directory). -2. **Prompt user iteratively** for each setting: - - LLM Model ID β†’ string - - Ollama Base URL β†’ string - - Ollama Context Size β†’ integer (parsed from the current value formatted as a string via `strconv.Atoi(ctxInputStr)`) - - Custom directories/files to ignore β†’ comma-separated list - - File extensions to ignore β†’ comma-separated list - - Documentation directory β†’ string -3. **Preserve existing values on empty input** β€” if the user presses Enter without typing anything, the previously loaded (or default) value is kept. -4. **Support reset semantics** β€” for list-type inputs, the literal strings `"clear"` or `"none"` wipe the list entirely; otherwise comma-splitting produces the new list. -5. **Error swallowing on read failures** β€” if stdin reading fails during any prompt (`bufio.NewReader(os.Stdin)` + `reader.ReadString('\n')`), the error is logged to stderr and execution continues using existing values (no rollback, no restart). -6. **Persist final config** β€” after all prompts complete, a single `config.Config` struct is constructed from user input + preserved state and written to disk via `config.SaveConfig`. The success message references a `.yaml` file path defined in the config package. +No concurrent access to shared state occurs within this file; no locking mechanism is needed. -### Key Business Rules +--- -- **Non-destructive defaults:** Existing configuration is always preferred over built-in constants; new values only replace what the user explicitly provided (or left as default). -- **Idempotent re-run:** Running `setup` again when a config already exists produces an incremental update rather than a full overwrite. -- **Graceful degradation on I/O errors:** Any stdin read failure does not abort setup; it silently falls back to existing state. +## Interactive Setup Flow (`cmd/setup.go`) -### Side Effects & Error Handling in `setup.go` +### Exported Function -**Disk reads (filesystem):** -- `os.Getwd()` β€” reads current working directory from filesystem. Returns error if unavailable; error is wrapped and propagated: `fmt.Errorf("failed to get current working directory: %w", err)`. -- `config.LoadConfig(repoRoot)` β€” reads a YAML config file from disk at `repoRoot`. If this fails, `existingCfg` remains `nil` (no panic). +| Function | Input(s) | Output | +|---|---|---| +| `RunSetupFlow(repoRoot string)` | `repoRoot string` | `error` | -**Disk writes (filesystem):** -- `config.SaveConfig(repoRoot, newCfg)` β€” writes the new configuration to disk. On error, wraps and returns: `"error saving configuration: %w"`. No retry or fallback. +No exported structs, interfaces, or methods are defined in this file. All other identifiers (`setupCmd`, `promptStringList`, `promptString`) and type references belong to imported packages or are unexported locals. -**Stdin reads (user input):** -- `bufio.NewReader(os.Stdin)` + `reader.ReadString('\n')` β€” called 6 times total across `promptString` and `promptStringList`: model ID prompt, base URL prompt, context size prompt, custom directories/files to ignore, custom file extensions to ignore, documentation directory. +### Business Rules Resolved by This File -**Stdout writes:** -- Welcome banner (`fmt.Println`) -- Each prompt label (e.g., `"Enter LLM Model ID [existing]:"`) -- Success confirmation: `Configuration successfully saved to local file.` +1. **Initial setup / reconfiguration** β€” supports both first-time configuration (no existing config) and reconfiguration (existing config present). Both paths use the same interactive prompts uniformly. +2. **Configuration persistence with defaults** β€” every configurable field has a built-in default value (`config.OllamaDefault*`, `config.DefaultDocsDir`, etc.). If no prior configuration exists, these defaults become initial values; if a config already exists, its current values serve as hints and are retained unless changed. +3. **LLM model configuration** β€” user must specify an LLM model ID and Ollama base URL for code analysis operations. Both fields accept empty input (falling back to existing/default), allowing partial reconfiguration. +4. **Context size with validation** β€” non-numeric or zero values are rejected, defaulting silently to the existing value rather than failing setup. +5. **Ignore list management** β€” directories/files/patterns to exclude from analysis can be set as comma-separated entries. Special keywords `clear` and `none` reset the list entirely. Existing ignores persist unless explicitly cleared. +6. **Prompt preservation strategy** β€” all four prompt-related fields (System Prompt, Module Synthesis Prompt, Architecture Prompt, File Fact Consolidation Prompt) follow a "preserve existing" rule: only updated if user provides non-empty input; otherwise current values remain unchanged. +7. **Atomic save** β€” after collecting all inputs, the entire configuration is written in one operation (`config.SaveConfig`). No partial writes occur. +8. **Error tolerance during setup** β€” input read errors and invalid conversions are swallowed with warning messages rather than aborting setup. Existing configuration values serve as safety nets throughout. -**Stderr writes:** -- On every `ReadString` failure, prints warning to stderr with the error value and falls back to existing/default values. No log level; just a single-line notice per call site. +### High-Level Algorithmic Flow -### Error Handling Patterns in `setup.go` +1. Load the current working directory; if it fails, propagate the error immediately (wrapped with `%w`). +2. Attempt to load an existing `.code-reducer.yaml` from that directory via `config.LoadConfig(repoRoot)`. Swallowed error β†’ treated as "no prior config". +3. For each configuration field: display a prompt with the current value in brackets as a hint β†’ collect user input via `reader.ReadString('\n')` on a `bufio.Reader(os.Stdin)` created in this function β†’ validate/convert β†’ fall back to existing/default on failure. +4. After all prompts complete, assemble a new `Config` struct combining any accepted inputs with preserved existing values. +5. Write the assembled config atomically via `config.SaveConfig(repoRoot, newCfg)`. On success, confirm completion; on write error, wrap and return as fatal exit for this flow. -| Failure point | Behavior | -|---|---| -| `os.Getwd()` | Wrapped into `fmt.Errorf("failed to get current working directory: %w", err)` and returned up the stack. | -| `config.LoadConfig(repoRoot)` | Nil-check only; if error occurs, existing values default to `OllamaDefault*` constants. No wrapping, no logging, no retry. Effectively swallowed. | -| `strconv.Atoi(ctxInputStr)` | Nil-check only; on parse failure or non-positive result, falls back to `existingNumCtx`. Swallowed silently. | -| `reader.ReadString('\n')` in prompt functions | Written to stderr as `"Warning: error reading input (%v), "`. Returns existing value (or empty for list). No propagation β€” caller never sees the error. | -| `config.SaveConfig(repoRoot, newCfg)` | Wrapped with `%w`. Returned up the stack via `RunE` on the cobra command, so cobra will print it to stderr and exit non-zero. This is the only path that surfaces errors to the user without extra wrapping. | +### Mutable State (`cmd/setup.go`) -### Notable Observations in `setup.go` +| Variable | Kind | Mutated After Initialization? | Notes | +|---|---|---|---| +| `setupCmd` (package-level) | `*cobra.Command` | No (assigned once in `init()`) | Internal fields (`Use`, `Short`, `Long`, `RunE`) set during construction. Not reassigned after init. | -- **No panic anywhere** β€” all I/O failures are handled (silently or with a warning). -- **Two error strategies:** some paths *propagate* (`Getwd`, `SaveConfig`), others *swallow and fallback* (`LoadConfig`, `Atoi`, prompt readers). This inconsistency is intentional: setup tolerates missing config, but treats save failures as fatal. -- **No retry logic** β€” `SaveConfig` is called once; if it fails, the entire flow aborts. -- **Stdin errors are not logged to a structured logger** β€” they go directly to stderr with a single `fmt.Fprintf`. +No concurrency primitives protect mutable state; no `sync.Mutex`, channels, atomic operations, or other synchronization primitives are present. --- -## Update Subcommand Registration (`update.go`) +## Error Handling Strategy -### Public Surface Area +### Pattern 1: Wrap-and-Return (Fatal) -This file exports **no public functions**, **no structs**, and **no interfaces**. +Used for unrecoverable setup failures β€” propagated up to Cobra's `RunE`: -### Internal Symbols - -| Symbol | Kind | Notes | -|---|------|---| -| `updateCmd` | `\*cobra.Command` | Local variable, not exported | -| `init()` | function | Lowercase; registered via Go's `init()` convention | - -### Business Domain Concept - -**Business Rule (stated):** When a project documentation system is updated incrementallyβ€”by scanning changed files since the last documented commitβ€”the wiki pages should be regenerated accordingly. - -**High-Level Algorithmic Flow:** -1. The user invokes an `update` subcommand on the root CLI (`RootCmd`). -2. Cobra parses and validates the command invocation. -3. The handler delegates to `executeCommand("update")`, which resolves the actual update logic (implementation not present in this file). - -**Notes:** This file only registers the CLI entry point; no business logic, validation, or data processing is contained here. - -### Side Effects & Error Handling in `update.go` - -| Concern | Status | +| Failure Point | Mechanism | |---|---| -| I/O (network/disk/DB) | Not present; delegated to external function (`executeCommand`) whose implementation is not present in this package boundary. | -| Error wrapping | None β€” raw pass-through of `executeCommand("update")` return value via `RunE`. | -| Sentinel errors | None created here. | -| Panic handling | Absent β€” if `executeCommand` panics, it propagates up unhandled (standard cobra behavior). | - ---- +| CWD lookup failure in `setup.go` | `fmt.Errorf("failed to get current working directory: %w", err)` | +| Config save failure | `error saving configuration: %w` with `%w` wrapping | +| Init re-run on initialized project (root.go) | Sentinel string error, no wrapping | -## Init Subcommand Registration (`init.go`) +### Pattern 2: Swallow-and-Fallback (Non-fatal) -### Public Surface Area of `cmd/init.go` +Used for user-input and transient I/O issues β€” existing values are preserved: -**Exported items:** None. +- `strconv.Atoi(ctxInputStr)` parse error β†’ falls back to `existingNumCtx` +- `promptString()` stdin read error β†’ prints warning to stderr, returns `existingVal` +- `promptStringList()` stdin read error β†’ prints warning to stderr, returns `existingList` unchanged +- `config.LoadConfig()` error β†’ treated as nil config; all fields default to built-in constants -### Business Domain Concept +### Pattern 3: Sentinel / Hard-Coded Messages (root.go) -CLI-driven documentation scaffolding tool (wiki generator). When the user runs `init`, the system scans the repository and generates an initial set of wiki markdown pages. This is a one-time setup operation for project documentation. - -### Algorithmic Flow - -1. User invokes the `init` subcommand on the CLI. -2. The command delegates to `executeCommand("init")`. -3. That function (defined elsewhere in this module) performs the repository scan and wiki page generation. - -**Note:** All substantive logic is deferred to `executeCommand`, which is not included in this file. This file serves only as registration/wiring for the subcommand within the cobra command hierarchy. +| Condition | Message | +|---|---| +| Init re-run on initialized project | `"the project has already been initialized"` | +| Update without prior initialization | explicit message | +| Missing config for run commands (non-TTY) | soft fail prompting user to run setup first | -### Side Effects & Error Handling in `init.go` +### Pattern 4: Event-Driven Output (root.go runner callback) -| Pattern | Used? | -|---|---| -| Wrap / aggregate errors | ❌ | -| Sentinel error | ❌ | -| Panic | ❌ | -| Pass-through delegation | βœ… β€” raw result of `executeCommand("init")` returned directly with no transformation. | +Errors are communicated via engine events (`engine.EventError`) routed to stderr rather than returned as error. Caller reads status through stdout/stderr stream instead of checking return value alone. The callback is side-effect only and does not propagate events back as errors. -### Mutable State & Concurrency in `init.go` +### Pattern 5: Context Cancellation (signal handler + `defer stop()`) -**No mutable state.** The only variable (`initCmd`) is initialized once at package scope and never reassigned. No concurrent access or modification occurs within this file. +Graceful shutdown on SIGINT/SIGTERM; runner receives cancelled context which the engine (outside this file) likely translates into an error return. If setup flow fails before reaching signal handler registration, cleanup cost is minimal. No panic observed in any of these files. --- -## Package-Level Summary +## Concurrency & Signal Handling -| Concern | Implementation Strategy | -|---|---| -| Command registration | Each subcommand file declares a local `\*cobra.Command` variable, registers it via `init()` (package-level), and wires its `RunE` callback to delegate to the shared `executeCommand(string) error` reference. | -| Configuration persistence | Handled by an external config package (`internal/config`). The `cmd` package only triggers the interactive flow in `setup.go` or reads flags in `root.go`. | -| Engine execution | Delegated to `engine.NewRunner.Run()`; event callbacks are defined inline in `root.go` for stdout/stderr routing. | -| Concurrency model | Sequential-only within this package boundary. Signal handling uses a cancellation context but does not protect shared mutable state. No channels, mutexes, or atomic types appear anywhere in the four files. | -| Error propagation strategy | Mixed: some paths propagate raw errors (`init`, `update`), others wrap and return sentinel messages (`root.go`'s git check), and still others swallow I/O failures silently with fallback values (`setup.go`). The only path that reaches cobra's default error printer is the final save failure in `setup.go`. | \ No newline at end of file +| Concern | Mechanism | Scope | +|---|---|---| +| Cooperative cancellation | `signal.NotifyContext` with `SIGINT`, `syscall.SIGTERM`; returned `ctx` and `stop()` manage lifecycle; deferring `stop()` cancels on return. | `cmd/root.go` | +| Signal-driven context propagation | Engine runner receives cancelled context which is translated into error by downstream code outside this package. | Cross-package boundary | + +No mutexes (`sync.Mutex`, `sync.RWMutex`) are used in any of these files. No channels or other synchronization primitives appear directly here. Only local/flag variables are mutated once at startup; no concurrent access to shared state occurs within these files, so no locking mechanism is needed. \ No newline at end of file diff --git a/wiki/modules/internal.md b/wiki/modules/internal.md deleted file mode 100644 index 2b310e9..0000000 --- a/wiki/modules/internal.md +++ /dev/null @@ -1,541 +0,0 @@ -# Architecture β€” Internal Packages - -## Module Responsibility & Data Flow - -The `internal` directory implements four orthogonal subsystems for the code analysis tool: **configuration resolution** (`config`), **pipeline orchestration** (`engine`), **sandbox enforcement** (`security`), and **filesystem/Git operations** (`tools`). Each subsystem has a single responsibility; cross-subsystem communication occurs only through the `*config.Config` value passed to engine entry points, which is read-only within `internal/engine`. - ---- - -## Package: config β€” Multi-Layer Configuration Resolution - -### Responsibility - -The `config` package implements a four-tier precedence pipeline for resolving all configurable values. Each layer overwrites prior values when present; missing layers are transparently skipped. Three properties are resolved independently (`ModelID`, `OllamaBaseURL`, `OllamaNumCtx`). The package also structures the analysis pipeline into named **extraction steps** (`ExtractionStep`), each carrying a distinct prompt template. - -All non-structural state (resolved config, local variables) is ephemeralβ€”scoped to function calls and returned as values. No package-level mutable state survives beyond initialization. Concurrency primitives are absent; the package is single-threaded in design. - -### Data Flow - -``` -DefaultExtractionSteps (init), DefaultIgnores, DefaultIgnoredExtensions - β”‚ - β–Ό -LoadConfig(cwd) ──► *Config{ModelID, OllamaBaseURL, OllamaNumCtx, ExtractionSteps, ...} - β”‚ β”‚ - β”‚ β”œβ”€β”€ defaults (hard-coded constants) - β”‚ β”œβ”€β”€ os.Getenv for CODE_REDUCER_MODEL_ID, OLLAMA_BASE_URL, OLLAMA_NUM_CTX - β”‚ └── strconv.Atoi on modelIdFlag, numCtxFlag (err==nil && n>0 required) - β”‚ - β–Ό -*Config returned to caller; no package-level mutation persists -``` - -### Types - -#### `ExtractionStep` - -```go -type ExtractionStep struct { - Name string // step identifier (e.g., "api-signatures", "business-logic") - Prompt string // prompt template applied to this phase during analysis -} -``` - -Instances are held in a package-level slice (`DefaultExtractionSteps`). The slice is immutable after declaration; callers receive copies or slices of it. - -#### `Config` - -```go -type Config struct { - ModelID string // resolved model identifier (last-wins: default β†’ YAML β†’ env β†’ flag) - OllamaBaseURL string // resolved base URL (default β†’ YAML β†’ env; CLI flag omitted) - OllamaNumCtx int // resolved context size, validated positive (default β†’ YAML β†’ env β†’ flag) - DocsDir string // path to the documents directory under analysis - ExtractionSteps []ExtractionStep // ordered list of extraction phases - Ignore []string // absolute paths excluded from traversal - IgnoreExtensions []string // file extensions excluded from traversal -} -``` - -### Constants and Defaults - -#### Environment Variable Keys - -| Constant | Value | Purpose | -|---|---|---| -| `CodeReducerModelIdEnvKey` | `"CODE_REDUCER_MODEL_ID"` | Overrides `Config.ModelID` when set. | -| `OllamaBaseUrlEnvKey` | `"OLLAMA_BASE_URL"` | Overrides `Config.OllamaBaseURL` when set. | -| `OllamaNumCtxEnvKey` | `"OLLAMA_NUM_CTX"` | Overrides `Config.OllamaNumCtx`; value must parse as a positive integer. | - -#### Default Values - -| Constant | Value | Purpose | -|---|---|---| -| `OllamaDefaultBaseURL` | `"http://localhost:11434"` | Fallback base URL when no prior layer supplies one. | -| `OllamaDefaultModelID` | `"ornith:9b"` | Default model identifier. | -| `OllamaDefaultNumCtx` | `8192` | Default context size. | - -#### Configuration File - -| Constant | Value | Purpose | -|---|---|---| -| `ConfigFileName` | `.code-reducer.yaml` | Filename resolved relative to the current working directory by `getConfigPath`. | - -### Package-Level Immutable Defaults - -Three slices are declared at package init and never mutated: - -```go -var DefaultIgnores []string // system-supplied ignore paths -var DefaultIgnoredExtensions []string // system-supplied ignored extensions -var DefaultExtractionSteps []ExtractionStep // ordered extraction phases -``` - -Callers may append to these slices externally, but the package itself never reassigns them. `SaveConfig` writes the current slice state back to disk; callers must mutate before save if they need custom defaults. - -### Public API Surface - -#### Path Resolution and Existence Check - -##### `getConfigPath(cwd string) (string, error)` - -Constructs the full path to `.code-reducer.yaml` by joining `cwd` with `ConfigFileName`. Returns a non-nil error only on I/O failure during construction; normal cases return `nil`. - -##### `ConfigExists(cwd string) bool` - -Returns whether the configuration file exists at the resolved path. Internally calls `os.Stat`; all errors (permission denied, I/O failures, etc.) are swallowed and treated as "does not exist". The function returns no error. - -#### Configuration Loading - -##### `LoadConfig(cwd string) (*Config, error)` - -Reads and parses the YAML configuration file at the resolved path. Error handling: -- File-read failure (`os.ReadFile`) β†’ returned to caller unchanged (pass-through). -- YAML parse failure β†’ wrapped as `"failed to parse yaml config: %w"`. - -Returns a non-nil `*Config` on success; the struct is initialized with zero values before parsing, so callers receive a populated object even when the file contains only valid empty mappings. - -##### `ResolveConfig(repoRoot, modelIdFlag, numCtxFlag string) (*Config, error)` - -Top-level entry point for full configuration resolution. Algorithm: -1. Calls `LoadConfig` to read the YAML file. If the error is a non-existent-file sentinel (`os.IsNotExist(err)`), it swallows and initializes an empty `Config{}`; all other errors propagate wrapped as `"failed to load configuration file:"`. -2. Layers defaults (hard-coded constants) over the loaded config. -3. Layers environment variable overrides using `os.Getenv` for `CODE_REDUCER_MODEL_ID`, `OLLAMA_BASE_URL`, and `OLLAMA_NUM_CTX`. Non-empty values are used; empty strings fall through. Env read errors are swallowed. -4. Applies CLI flags (`modelIdFlag`, `numCtxFlag`). The flag value is parsed via `strconv.Atoi`; only `err == nil && n > 0` is acceptedβ€”invalid or non-positive values cause the prior layer's value to be retained silently. - -Returns a fully resolved `*Config`. Errors are either pass-through (from `LoadConfig`) or wrapped with `"failed to load configuration file:"`. Notable: if YAML is malformed, the parse error wraps and propagates; otherwise resolution always succeeds for missing files. - -#### Configuration Persistence - -##### `SaveConfig(cwd string, cfg *Config) error` - -Serializes the config struct back to disk using `os.WriteFile` with `0600` permissions. Error handling: -- Write success β†’ returns `nil`. -- Parse failure (marshal YAML) β†’ wrapped as `"failed to marshal yaml: %w"`. -- File-write failure β†’ wrapped as `"failed to write config file: %w"`. - -Writes directly (no temp-file + rename); a partial write can leave a truncated file on disk. Permission normalization is implicitβ€”existing files with different permissions will fail the write. - ---- - -## Package: engine β€” Code Documentation Pipeline Engine - -### Responsibility - -The `internal/engine` package implements a Map-Reduce pipeline that generates and maintains technical documentation for a Go codebase using Ollama-compatible LLM inference endpoints. The engine produces three artifacts per run: `architecture.md` (global overview), `quickstart.md` (onboarding guide), and per-module API summaries under `modules/`. Execution is gated by a repository-level lock to prevent concurrent runs against the same root, and state persistence across invocations is handled by an on-disk metadata cache. - -### Data Flow - -``` -security.EnsureGitignoreHasLockfile(repoRoot) ──► security.AcquireLock(repoRoot) ──► defer Unlock() - β”‚ β”‚ - β–Ό β–Ό -*LLMClient from config.ModelID, BaseURL, NumCtx RunInit(ctx, repoRoot, cfg, onEvent) - β”‚ β”‚ - β–Ό β–Ό -client.RunInit / client.RunUpdate β†’ synthesizeNode (bottom-up tree reduction) β†’ CallLLM per step/chunk - β”‚ - β–Ό -tools.WriteFileSafely for architecture.md, quickstart.md, modules/*.md -``` - -### Pipeline Orchestration - -#### Runner Entry Point (`runner.go` β€” `Runner.Run`) - -The public entry point for all pipeline invocations. Responsibilities: repository isolation, concurrency guard acquisition, LLM client construction from config, and mode-gated dispatch to the appropriate pipeline method. - -``` -Run(ctx, repoRoot, mode string, onEvent func(Event)) error - β”œβ”€ ensure .gitignore lockfile entry (best-effort; failure logged as warning) - β”œβ”€ acquire repository-level lock via security.AcquireLock(repoRoot) - β”‚ └─ defer lock.Unlock() β€” released on any return path including errors - β”œβ”€ build *LLMClient from r.cfg.ModelID, BaseURL, NumCtx - β”œβ”€ switch mode: - β”‚ case "init": client.RunInit(ctx, repoRoot, r.cfg, onEvent) - β”‚ case "update": client.RunUpdate(ctx, repoRoot, r.cfg, onEvent) - β”‚ default: return fmt.Errorf("unsupported mode: %s", mode) -``` - -The `cfg *config.Config` field is read-only within this file; all mutations to the runner's configuration originate before construction. The lock returned by `security.AcquireLock(repoRoot)` is a local variable with deferred cleanupβ€”its internal type and mutual exclusion semantics are not observable from this source alone. - -#### Orchestrator Pipelines (`orchestrator.go` β€” `RunInit`, `RunUpdate`) - -##### Full Run (`RunInit`) - -Executes an all-or-nothing regeneration regardless of cache state: - -1. **Setup** β€” Load `.gitignore` via `tools.LoadGitignore(repoRoot)`; merge with user ignore list; load metadata cache from disk via `cache.loadMetadataCache()`. Failures logged as warnings, not returned. -2. **Map Phase** β€” Discover code files via `tools.DiscoverCodeFiles(...)`, compute SHA-256 per file via `computeSHA256(repoRoot, f)`, build directory tree via `buildTree(codeFiles)`. Hash computation errors silently skipped in loops; only stored if hash computed successfully. -3. **Mark All Affected** β€” Recursively flag every node as affected (init starts fresh). -4. **Reduce Phase** β€” Call `synthesizeNode` bottom-up on the tree to produce a root summary string and per-module summaries via LLM calls through `c.CallLLM`. -5. **Generate Standard Docs** β€” Produce `architecture.md` and `quickstart.md` from the root summary via two additional `CallLLM` invocations: one for architecture, one for quickstart. Writes via `tools.WriteFileSafely`; errors wrapped with context (e.g., `"failed to write quickstart.md: %w"`). -6. **Update AGENTS.md** β€” Read existing file via `tools.ReadFileSafely(repoRoot, "AGENTS.md")`. If missing, create with guidelines header. If present but does not contain "AI Agent Guidelines", append separator + new content block. Only writes when content actually differs. Errors: if read fails but write succeeds β†’ `"failed to write AGENTS.md: %w"`; if both fail β†’ wrapped errors chain. -7. **Teardown** β€” Persist metadata cache via `saveMetadataCache(repoRoot, docsDir, cache)`. Failure logged as warning, function always returns nil at end regardless of this failure. - -##### Incremental Run (`RunUpdate`) - -Executes only re-generations for directories whose contents actually changed: - -1. **Setup** β€” Load pipeline state from metadata cache (same path as init). -2. **Map Phase** β€” Discover code files, hash each via `computeSHA256`, build tree. -3. **Change Detection** β€” Compare current file hashes against cached hashes in `cache.Files`: - - New file in codebase but absent from cache β†’ `FileChange{Status: "Added"}` - - Existing file with different hash β†’ `FileChange{Status: "Modified"}` - - Cached file no longer in codebase β†’ `FileChange{Status: "Deleted"}` (also removed from cache) -4. **Prune Stale Modules** β€” Remove any module summaries whose directory is no longer active, delete their markdown files on disk via `os.Remove(absModuleFile)` (error assigned to `_`, never logged). -5. **Propagate Affected** β€” Run tree-aware propagation (`propagateAffected`/`determineAffected`) to mark directories affected by changes. Errors from these calls are discarded without capture. -6. **Early Exit** β€” If zero directories are affected β†’ pipeline ends with "up to date" status, no further synthesis occurs. -7. **Conditional Reduce** β€” If root-level change detected or global docs don't exist on disk (`os.Stat(absArch)`/`os.Stat(absQs)`), run full synthesis; otherwise skip and reuse existing `architecture.md`/`quickstart.md`. -8. **Teardown** β€” Save updated cache via `saveMetadataCache(repoRoot, docsDir, cache)`. - -### Directory Tree Construction & Affected Propagation (`tree.go`) - -#### Type Definitions - -| Type | Fields | Purpose | -|------|--------|---------| -| `FileChange` | `Path string`, `Status string` | Models a file operation with path and status (`Added`, `Modified`, `Deleted`). Consumed by `determineAffected`. | -| `DirNode` | `Path string`, `Files []string`, `Children map[string]*DirNode` | Hierarchical tree node holding files at that level and child directories in its subtree. Produced by `buildTree`. | - -#### Functions (all unexported) - -- **`buildTree(codeFiles)`** β€” Converts a flat list of file paths into nested `*DirNode` structure, preserving directory nesting by splitting on `/`. Appends to `DirNode.Files`; populates `DirNode.Children` via map assignment. -- **`propagateAffected(tree, changeset)`** β€” Traverses the tree recursively: if any direct child of the current node is in the changeset, mark it affected. Accumulates affected status upward through ancestors. Returns mutated `affectedDirs`. -- **`determineAffected(tree, changeset)`** β€” Computes module path for each directory via `ToSafeMarkdownFilename`, checks existence on disk via `os.Stat(absModulePath)`, reads metadata cache (`cache.Modules[n.Path] == ""`), marks node affected if any check fails. Only `os.IsNotExist` is acted on; all other errors are silently dropped. - ---- - -## Package: engine β€” LLM Client Abstraction & Response Parsing - -### Type: `LLMClient` - -```go -type LLMMClient struct { - ModelID string - BaseURL string - NumCtx int - HTTPClient *http.Client // default transport, no custom config (no proxy/redirect/TLS) -} -``` - -Constructed via `NewLLMClient(modelID, baseURL, numCtx)` with a 10-minute HTTP timeout. Fields set once; never reassigned within any method. - -### Methods on `*LLMClient` - -#### `CallLLM(ctx context.Context, systemPrompt string, messages []Message, jsonFormat bool) (string, error)` - -Full lifecycle: prepends system prompt as first message with role `"system"` β†’ serializes into Ollama `/api/chat` JSON payload with model ID and context window size β†’ non-streaming HTTP POST β†’ response body fully read on 200 OK or truncated at 1 KB via `io.LimitReader(resp.Body, 1024)` on non-OK status. Returns only assistant content string; returns empty string on failure. - -**Error behavior:** - -| Failure path | Error type | Wrapping strategy | -|---|---|---| -| JSON marshal fails in request prep | `fmt.Errorf("failed to marshal request: %w", err)` | Wrapped with `%w` | -| `http.NewRequestWithContext` fails | raw error from `NewRequestWithContext` | **Not wrapped** β€” passed through as-is | -| HTTP Do (network-level failure) | raw error from `c.HTTPClient.Do(req)` | **Not wrapped** | -| Body read fails on 200 OK path | raw error from `io.ReadAll` | **Not wrapped** | -| JSON unmarshal fails on response | `fmt.Errorf("failed to parse response: %w", err)` | Wrapped with `%w` | -| Non-OK status (body truncated at 1 KB) | `fmt.Errorf("ollama api error: status %d, response: %s", ...)` | New error constructed; body discarded if read fails | - -#### `GetBaseSystemPrompt() string` - -Returns the fixed base role definition and defensive guidelines inherited by every LLM call regardless of task type. - -#### `GetDefaultSystemPrompt(command string) string` - -Appends command-specific instructions based on the command string: two named variants exist (`module_synthesis`, `architecture`); any other value receives only the base layer. - -### Response Parsing Layer (`json_parser.go`) - -Internal helper `extractBalancedJSON` is lowercase and excluded from public surface area. - -#### Business Rules - -1. Markdown code fences (` ``` `) surrounding JSON must be stripped before parsing β€” handled by `StripOuterMarkdownFence`. -2. Balanced bracket matching required β€” strings containing `{`, `[`, `}`, `]` literals (including escaped characters like `\n`) must not break extraction logic β€” handled by internal rune-level scanner with escape state tracking inside string literals. -3. When multiple balanced JSON candidates exist in a single response, each must be attempted individually until one succeeds, or the entire trimmed string is tried as a final fallback β€” handled by `CleanJSONResponse`/`UnmarshalJSONResponse`. - -#### Functions - -| Function | Behavior | Error surface | Swallowed? | -|---|---|---|---| -| `StripOuterMarkdownFence(s string) string` | Strips surrounding markdown fences; falls back to trimmed input if none match | None β€” returns string only | N/A (no error return) | -| `CleanJSONResponse(s string) string` | Iterates stripped content, calls `extractBalancedJSON`, returns first successful candidate or original trimmed input unchanged | None β€” returns string only | Yes β€” unbalanced JSON inside response is swallowed silently | -| `UnmarshalJSONResponse(s string, target interface{}) error` | Multi-layered attempt: (1) collect balanced candidates, try each into target; keep last error in `lastErr`; (2) if any succeeded, return nil; otherwise try entire trimmed string; (3) terminal errors β€” `"failed to unmarshal JSON candidates: %w"` or `"no valid JSON found in response"` | Explicit (`error`) | Partial parse failures collected and re-emitted via `%w` on first path; "no valid JSON" covers second path | - ---- - -## Package: engine β€” Synthesis Engine & Chunking - -### Context-Aware Chunking (`chunking.go` β€” all unexported) - -The system processes extracted code/fact items in batches constrained by LLM context limits (`NumCtx * 3`). Three distinct processing modes exist: - -#### Module Synthesis (`reduceInChunks`) - -Synthesizes architectural-level nodes into architecture descriptions using a `"module_synthesis"` system prompt. Each batch represents evidence contributing to understanding the module's design intent. Response passes through `StripOuterMarkdownFence`. Errors wrapped with `"LLM error during synthesis: %w"`. - -#### File Fact Consolidation (`reduceFileFacts`) - -Consolidates extracted facts about a specific step within a file with deduplication and merging. System prompt built locally by concatenating `c.GetBaseSystemPrompt()` + fixed role string. Response also passes through `StripOuterMarkdownFence`. Errors wrapped with `"LLM error during file fact consolidation: %w"`. - -#### Recursive Convergence (`reduceItems`) - -Core business rule: reduce items in batches β†’ collect intermediate results β†’ recursively apply reduction until everything converges into a single output string. Multi-pass approach ensures information is progressively synthesized rather than lost across multiple LLM calls. Termination guaranteed when all data fits within one batch (short-circuits at top with `len(batches) == 1`). Recursion terminates because: -1. Single-batch case short-circuits at top (`len(batches) == 1`) -2. Recursive step processes each batch through itself again with same `maxChars` - -**Graceful Truncation Policy:** When content exceeds the context window, single oversized items get truncated with `\n...[truncated]` marker appended; when multiple items exceed limits collectively, each gets capped at `maxChars / len(items)` characters before being processed as one batch. Infinite-loop guard: if batch size equals item count for all items (can't merge), falls back to per-item truncation using `allowedPerItem = maxChars / len(items)`. Still silent β€” no error returned. - -#### Overlapping Chunk Support (`chunkTextWithOverlap`) - -Utility for splitting raw text into overlapping chunks to maintain context continuity between adjacent segmentsβ€”useful when feeding source material directly into LLMs without pre-processing through the chunking logic. No I/O. Silent swallow: `maxRunes <= 0` β†’ returns full text as single chunk; `overlapRunes >= maxRunes` β†’ clamps to half without warning; final partial chunk reaching end of runes β†’ breaks loop with no error or truncation marker (unlike `reduceItems`). - -### Hierarchical Module Synthesis (`synthesize.go` β€” `synthesizeNode`, all unexported) - -Performs recursive synthesis of directory structures into hierarchical summariesβ€”transforming raw source code and subdirectory metadata into consolidated documentation at each level of a module tree. - -**Algorithmic Flow:** -1. **Context Check β†’ Cache Reuse Gate**: If current node not in `affectedDirs` AND cached module summary exists, return immediately without processing. Short-circuits unaffected directories. -2. **Recursive Child Synthesis (Bottom-Up)**: Sort children alphabetically, recursively call `synthesizeNode` on each child directory; results collected into map keyed by child name. -3. **File Processing Loop**: For each file in current node: hash-first cache lookup (if precalculated hash exists and matches cached SHA256 β†’ reuse cached facts without reading disk); read + hash fallback otherwise (read via `tools.ReadFileSafely`, compute SHA256, update cache entry if already existed with that hash); extraction pipeline on cache miss (chunk content using dynamic limit derived from `c.NumCtx Γ— 4 characters Γ— 0.75` with minimum of 512 tokens and max overlap cap; for each chunk iterate over every step in `cfg.ExtractionSteps`, sending to LLM with system prompt built from base + step-specific prompt; consolidate all chunks for that step via `reduceFileFacts`; append consolidated result under step's name heading). -4. **Component Assembly**: After processing files, append any non-empty child summaries to components list. If no components exist at all (no files AND no children), clear module cache and return empty string. -5. **Final Synthesis & Persistence**: Pass assembled components to `reduceInChunks` for directory path; store result in both memory cache (`cache.Modules[node.Path]`) and on disk at `//modules/` via `tools.WriteFileSafely`; return final summary string. - -**Domain concepts:** Hierarchical module synthesis (bottom-up: files contribute facts, subdirectories contribute their own summaries, parent combines both); context-aware truncation (file content limits scale with `c.NumCtx`, reserving 75% of tokens for file data and 25% for prompts/output); two-level caching (files cache individual facts keyed by SHA256, modules cache full synthesized summaries keyed by path; reuse only when node not affected); multi-step extraction pipeline (each file passes through configurable sequence of LLM steps, each producing independent fact set consolidated before combining with other files' results). - -**Error handling in `synthesizeNode`:** - -| Source | Pattern | Propagation | -|---|---|---| -| Context cancel | Direct return at top and inside per-file loop (`if err := ctx.Err(); err != nil { return "", err }`) | Upward, no wrapping | -| LLM call failure | Wrapped: `"LLM error extracting %s for %s: %w"` with step name + file path | Upward | -| File read failure | Logged via `logEvent("status", ...)`, skipped silently; no hash/facts/anything added to components | None (swallowed) | -| Module write failure | Wrapped: `"failed to write module documentation for %s: %w"` with dir path | Upward | -| Recursive child error | Propagated as-is (`return "", err`) | Upward | -| Empty components | Returns `("", nil)`, clears `cache.Modules[node.Path] = ""` | None (clean return) | - ---- - -## Package: engine β€” Filesystem State Management & Utilities - -### Type: `MetadataCache` - -```go -type MetadataCache struct { - Files map[string]FileCacheEntry // virtual path β†’ cached SHA256 + facts - Modules map[string]string // module identifier β†’ summary string -} -``` - -Both maps initialized with `make()` when nil; populated during load/save lifecycle; serialized to disk. No in-memory locking applied within this file. - -### Functions (all unexported except `IsInitialized`) - -#### Core Concept: Filesystem-Based State Cache - -Persists an in-memory cache to disk so expensive computations can be avoided across runs. Stored as `docsDir/.metadata.json`, holds two pieces of state: -- **File-level metadata** (`Files map`): maps virtual paths (filenames) to cached SHA256 hashes plus associated "facts" strings. -- **Module-level metadata** (`Modules map`): maps module identifiers to string values. - -#### Functions - -| Function | Direction | Path constructed | Notes | -|---|---|---|---| -| `loadMetadataCache()` | Read | `docsDir/.metadata.json` | Attempts to read JSON; if absent, returns freshly initialized empty cache (not error); if present but unparseable, errors. Null maps normalized to empty on load. | -| `IsInitialized(repoRoot, docsDir string) bool` | Read (existence probe) | `docsDir/.metadata.json` | Probes for existence of `.metadata.json`. Successful read β†’ true; any non-nil error β†’ false. N/A for missing file (returns false). | -| `saveMetadataCache()` | Write | `docsDir/.metadata.json` | Serializes struct back to JSON with indentation, writes atomically via `tools.WriteFileSafely`. Returns raw errors from marshal/write unchanged. | -| `computeSHA256(virtualPath)` | Read (virtual path) | caller-supplied virtual path | Reads file by virtual path, runs SHA-256 on content, returns hex-encoded digest. No disk I/O within this file; pure in-memory (`crypto/sha256`). | - -**Error handling patterns:** - -| Function | On missing cache file | On generic read/write error | On JSON parse error | -|---|---|---|---| -| `loadMetadataCache` | Returns empty initialized `*MetadataCache`, nil error | Wraps with `"failed to read metadata cache: %w"` | Wraps with `"failed to unmarshal metadata cache: %w"` | -| `IsInitialized` | N/A (returns false) | N/A β€” any non-nil error β†’ false | β€” | -| `saveMetadataCache` | β€” | Returns raw error from `tools.WriteFileSafely` | Returns raw error from `json.MarshalIndent` | -| `computeSHA256` | β€” | Returns raw error from `tools.ReadFileSafely` | N/A (no parsing) | - -### Function: `ToSafeMarkdownFilename(modulePath string) string` - -Pure string transformation with no business logic or complex algorithmic flow. Maps a module path to a safe filename for markdown documentation. - -**Conversion pipeline:** -1. Replace every `/` character in input `modulePath` with `_`. -2. If result is empty or equals `"."`, substitute with `"root"`. -3. Append `.md` and return. - -**Domain concept:** Module-to-document routing β€” given a module identifier, produce corresponding markdown file path. Assumes caller already owns consistent `modulePath` convention; this utility only adapts that path for filesystem-safe filenames. - ---- - -## Package: security β€” Sandbox Enforcement & Process Locking - -### Responsibility - -The `security` package encapsulates three orthogonal concerns for code-reducer processes: **sandbox enforcement** (preventing path traversal beyond the repository boundary), **process-level mutual exclusion** via file-based locks, and **operational hygiene** (keeping lock state out of version control). No networking, database access, or external process communication occurs within this package. - -All functions are pure-Go with no global mutable state. Every observable side effect is local to a single `*SimpleLock` receiver instance or the calling goroutine's stack. Errors follow a strict wrapping convention: security-critical violations and actionable sentinel messages return unwrapped plain strings; standard-library errors use `%w`; best-effort cleanup paths swallow non-fatal I/O failures silently. - -**Data flow** proceeds in two independent tracks that share no state except through the repository root path string: - -1. **Path resolution track:** `SafeResolve` β†’ called by `AcquireLock` to canonicalize the lock file location before attempting acquisition. -2. **Lock lifecycle track:** `AcquireLock` β†’ returns a `*SimpleLock` receiver β†’ caller uses it for work β†’ calls `Unlock()` for release β†’ `EnsureGitignoreHasLockfile` runs during cleanup to append the lock filename to `.gitignore`. - -### Types - -#### `SimpleLock` - -File-based process lock with internal mutex protection. The struct is constructed by `AcquireLock`; once acquired, `lockPath` and `file` are set exactly once under mutex scope, then only modified during `Unlock()`. - -| Field | Type | Semantics | -|-------|------|-----------| -| `lockPath` | `string` | Absolute path to the `.code-reducer.lock` file. Set in constructor; zero-valued if acquisition failed or was never called. | -| `file` | `*os.File` | Opened lock file handle. Acquired with `O_CREATE\|O_EXCL`; set to `nil` inside `Unlock()`. | -| `mu` | `sync.Mutex` | Per-instance mutex protecting `closed` and the pointer-clear of `file` during unlock. | -| `closed` | `bool` | Transition flag: `false` while locked, flips to `true` once `Unlock()` completes its write operations (under mutex). | - -### Public API Surface - -#### Functions - -##### `SafeResolve(repoRoot, inputPath string) (string, error)` - -Computes the absolute path of an input relative to a repository root. The resolution is symlink-aware: ancestor components are evaluated through their physical targets via `filepath.EvalSymlinks` before being walked upward until one succeeds. The final result must lie strictly inside the resolved rootβ€”any escape beyond it returns an unwrapped string error indicating a security violation. - -##### `AcquireLock(repoRoot string) (*SimpleLock, error)` - -Resolves the lock file path through `SafeResolve`, then attempts atomic creation with `O_CREATE\|O_EXCL`. On success, writes the process PID to the newly created file and returns a populated `*SimpleLock`. If the file already exists (anewer process holds it), returns an unwrapped sentinel string describing that another process has acquired the lock. - -##### `EnsureGitignoreHasLockfile(repoRoot string) error` - -Reads `.gitignore`; if present, appends `# Code-Reducer Lockfile\n.code-reducer.lock\n` only when not already contained in the file. If `.gitignore` does not exist, creates it with append semantics (`O_APPEND\|O_CREATE\|O_WRONLY`). Errors from reading or writing are wrapped; non-existence of `.gitignore` is treated as expected and returned as `nil`. - -#### Methods - -##### `(l *SimpleLock).Unlock() error` - -Idempotent release: acquires the internal mutex, closes the file handle (discarding any close-error), removes the lock file from disk via best-effort `os.Remove`, sets `closed = true`, then releases the mutex. A second call is safe and returns `nil`. Swallowed errors for cleanup operations are discarded; removal failures that occur after a successful close surface as wrapped errors only in the specific narrow case where both close-succeeded-and-remove-failedβ€”but this path effectively swallows non-`IsNotExist` remove errors in practice. - -### Business Rules & Domain Concepts - -#### Repository Boundary Enforcement - -All internal paths must resolve strictly within `repoRoot`. Any resolution that escapes beyond it returns a standalone string error: `"security violation: path traversal detected: %q"`, never wrapped. This is the one unwrapped class of non-sentinel errors in the packageβ€”security-critical, so callers detect it via exact match or `strings.Contains` rather than type assertion. - -#### Symlink-Aware Path Resolution - -Ancestors are evaluated through their physical targets before path reconstruction. This prevents traversal attacks that use symlinked directories to bypass the sandbox while preserving logical input structure for legitimate uses. Implementation: `filepath.EvalSymlinks(absRoot)` resolves the root once; subsequent ancestor walks use `os.Lstat` with a fallback to `EvalSymlinks` on each component until one exists, then rebuild upward from that point. - -#### Process-Level File Locking - -The lock file is `.code-reducer.lock`. Acquisition uses `O_EXCL` for atomic creationβ€”no two processes can hold the lock simultaneously on the same filesystem. On success, the PID of the acquiring process is written as a single-line string (`fmt.Sprintf("%d\n", os.Getpid())`). The lockfile content therefore serves as both a record and a mechanism for stale detection: if another process exits without releasing its lock, a new acquisition attempt detects the existing file and reports an actionable error instructing manual cleanup. - -#### Stale Lock Detection - -Intentional behaviorβ€”the system does not attempt to detect or kill the holding process. The caller receives an unwrapped string describing that the lock is held by another process and must clean up manually. This keeps the package non-invasive; it never interacts with external processes. - -#### Automated Gitignore Integration - -The lockfile path is appended to `.gitignore` during unlock so it does not enter version control. Idempotent: if a line containing `code-reducer.lock` already exists, no duplicate entry is written. The lock filename itself is the only thing tracked; comment prefix provides human readability without affecting git behavior. - ---- - -## Package: tools β€” Filesystem & Git Operations - -### Responsibility - -This module provides two complementary interfaces for repository-aware operations: safe file I/O with TOCTOU race mitigation, and Git process abstraction for executing commands within the working tree. Both modules operate on a single root path (`repoRoot`) without shared mutable state, and neither uses concurrency primitives β€” all functions are goroutine-safe by construction. - -### `file_tools.go` β€” Safe File I/O & Discovery - -#### Core Functions - -| Function | Input | Output | -|---|---|---| -| [`ReadFileSafely`](file:///path/to/file_tools.go#L15-L64) | `repoRoot`, `virtualPath` `string` | `[]byte`, `error` | -| [`WriteFileSafely`](file:///path/to/file_tools.go#L70-L132) | `repoRoot`, `virtualPath` `string`; `content` `[]byte` | `error` | -| [`LoadGitignore`](file:///path/to/file_tools.go#L138-L164) | `repoRoot` `string` | `[]string`, `error` | -| [`ShouldIgnoreFile`](file:///path/to/file_tools.go#L170-L236) | `repoRoot`, `relPath` `string`; `gitIgnore` `*ignore.GitIgnore`; `ignoredExtensions` `[]string` | `bool` | -| [`DiscoverCodeFiles`](file:///path/to/file_tools.go#L242-L291) | `repoRoot` `string`; `ignores`, `ignoredExtensions` `[]string` | `[]string`, `error` | -| [`IsBinaryFile`](file:///path/to/file_tools.go#L297-L315) | `path` `string` | `bool` | - -#### TOCTOU Race Mitigation Pattern - -Both `ReadFileSafely` and `WriteFileSafely` implement a Time-Of-Check-Time-Of-Use safety pattern: - -1. Resolve the virtual path to an absolute safe path via `SafeResolve` -2. Open the file descriptor first -3. Perform `lstat` on the resolved path (no symlink following) -4. Perform `fstat` on the open file descriptor (follows symlinks) -5. Verify that `lstat` and `fstat` reference the same inode β€” mismatch indicates a symlink race between opening and checking - -Both operations reject any symlink detection as a security violation. On success, `ReadFileSafely` reads all content via buffered I/O; on success, `WriteFileSafely` truncates to zero first (safe only because symlinks were ruled out), then writes the provided content. - -#### Error Handling Contract - -| Function | Wrapped Errors (`%w`) | Swallowed | Sentinel Unwrapped | -|---|---|---|---| -| `ReadFileSafely` | Yes, for every I/O error | No | Symlink race check | -| `WriteFileSafely` | Yes, for every I/O error | No | Symlink race check | -| `LoadGitignore` | No | Missing file β†’ `(nil, nil)` | None | -| `ShouldIgnoreFile` | No | `SafeResolve` err β†’ returns `true` | None | -| `DiscoverCodeFiles` | No | Extensive β€” walk callback errors swallowed at each node | None | -| `IsBinaryFile` | No | All open/read errors β†’ `false` | None | - -#### File Discovery Algorithm (Multi-Layer Ignore Rules) - -A file is considered "ignored" if it matches **any** of these conditions: - -- Matches a compiled gitignore pattern list -- Has a path component starting with `.` or ending in `.egg-info` -- Has an extension matching the user-provided ignored extensions list (case-insensitive, supports both `.` prefix and suffix forms) -- Is detected as binary by reading up to 1024 bytes and scanning for null byte (`\x00`) - -If any single check returns true, the file is excluded. This is a **union** of ignore conditions. - -The primary business rule is to recursively walk the repository root and discover high-signal source code files while filtering out noise: build artifacts, dependency directories, output files, dot-prefixed hidden paths, `.egg-info` markers, binary files, and any user-defined ignore patterns (via gitignore compilation). - -#### Binary Detection Rule - -A file is considered binary if its first 1024 bytes contain a null byte (`\x00`). Files that cannot be opened are treated as non-binary (returns false), avoiding the loading of entire files into memory for classification. - -### `git_tools.go` β€” Git Process Abstraction - -#### Core Functions - -| Function | Input | Output | -|---|---|---| -| [`RunGit`](file:///path/to/git_tools.go#L15-L64) | `repoRoot string`, variadic `args ...string` | `(string, error)` | -| [`VerifyGitRepo`](file:///path/to/git_tools.go#L70-L89) | `repoRoot string` | `error` | - -#### Execution Model - -Both functions spawn the `git` binary via `exec.Command("git", ...)` with `--no-pager`, restricting execution to the provided root directory. Both capture stdout and stderr into buffers synchronously. No standard library packages beyond `bytes`, `fmt`, `os/exec`, `strings` are imported. - -#### Error Handling Contract - -| Function | Failure Path | Behavior | -|---|---|---| -| `RunGit` | Exit code β‰  0 | Wraps error: `git command failed: %v, stderr: %s`. Returns `(string, error)` with stdout and wrapped error. No panic \ No newline at end of file diff --git a/wiki/modules/internal_config.md b/wiki/modules/internal_config.md deleted file mode 100644 index 7328aeb..0000000 --- a/wiki/modules/internal_config.md +++ /dev/null @@ -1,121 +0,0 @@ -# `internal/config` β€” Package Documentation - -## Responsibility and Data Flow - -The package implements a **multi-layer configuration resolution** pipeline for the code analysis tool. Configuration values flow through four precedence tiers: hard-coded defaults β†’ YAML file (`~/.config/code-reducer.yaml`) β†’ environment variables β†’ CLI flags. Each layer overwrites prior values when present; missing layers are transparently skipped. - -Three properties are resolved independently: `ModelID`, `OllamaBaseURL`, and `OllamaNumCtx`. The package also structures the analysis pipeline into named **extraction steps** (`ExtractionStep`), each carrying a distinct prompt template. These steps can be customized per project via YAML or wholesale replaced by CLI flag. - -All non-structural state (resolved config, local variables) is ephemeralβ€”scoped to function calls and returned as values. No package-level mutable state survives beyond initialization. Concurrency primitives are absent; the package is single-threaded in its design. - -## Types - -### `ExtractionStep` - -```go -type ExtractionStep struct { - Name string // step identifier (e.g., "api-signatures", "business-logic") - Prompt string // prompt template applied to this phase during analysis -} -``` - -Instances are held in a package-level slice (`DefaultExtractionSteps`). The slice is immutable after declaration; callers receive copies or slices of it. - -### `Config` - -```go -type Config struct { - ModelID string // resolved model identifier (last-wins: default β†’ YAML β†’ env β†’ flag) - OllamaBaseURL string // resolved base URL (default β†’ YAML β†’ env; CLI flag omitted) - OllamaNumCtx int // resolved context size, validated positive (default β†’ YAML β†’ env β†’ flag) - DocsDir string // path to the documents directory under analysis - ExtractionSteps []ExtractionStep // ordered list of extraction phases - Ignore []string // absolute paths excluded from traversal - IgnoreExtensions []string // file extensions excluded from traversal -} -``` - -## Constants and Defaults - -### Environment Variable Keys - -| Constant | Value | Purpose | -|---|---|---| -| `CodeReducerModelIdEnvKey` | `"CODE_REDUCER_MODEL_ID"` | Overrides `Config.ModelID` when set. | -| `OllamaBaseUrlEnvKey` | `"OLLAMA_BASE_URL"` | Overrides `Config.OllamaBaseURL` when set. | -| `OllamaNumCtxEnvKey` | `"OLLAMA_NUM_CTX"` | Overrides `Config.OllamaNumCtx` when set; value must parse as a positive integer. | - -### Default Values - -| Constant | Value | Purpose | -|---|---|---| -| `OllamaDefaultBaseURL` | `"http://localhost:11434"` | Fallback base URL when no prior layer supplies one. | -| `OllamaDefaultModelID` | `"ornith:9b"` | Default model identifier. | -| `OllamaDefaultNumCtx` | `8192` | Default context size. | - -### Configuration File - -| Constant | Value | Purpose | -|---|---|---| -| `ConfigFileName` | `".code-reducer.yaml"` | Filename resolved relative to the current working directory by `getConfigPath`. | - -## Package-Level Immutable Defaults - -Three slices are declared at package init and never mutated: - -```go -var DefaultIgnores []string // system-supplied ignore paths -var DefaultIgnoredExtensions []string // system-supplied ignored extensions -var DefaultExtractionSteps []ExtractionStep // ordered extraction phases -``` - -Callers may append to these slices externally, but the package itself never reassigns them. `SaveConfig` writes the current slice state back to disk; callers must mutate before save if they need custom defaults. - -## Public API Surface - -### Path Resolution and Existence Check - -#### `getConfigPath(cwd string) (string, error)` - -Constructs the full path to `.code-reducer.yaml` by joining `cwd` with `ConfigFileName`. Returns a non-nil error only on I/O failure during construction; normal cases return `nil`. - -#### `ConfigExists(cwd string) bool` - -Returns whether the configuration file exists at the resolved path. Internally calls `os.Stat`; all errors (permission denied, I/O failures, etc.) are swallowed and treated as "does not exist". The function returns no error. - -### Configuration Loading - -#### `LoadConfig(cwd string) (*Config, error)` - -Reads and parses the YAML configuration file at the resolved path. Error handling: -- File-read failure (`os.ReadFile`) β†’ returned to caller unchanged (pass-through). -- YAML parse failure β†’ wrapped as `"failed to parse yaml config: %w"`. - -Returns a non-nil `*Config` on success; the struct is initialized with zero values before parsing, so callers receive a populated object even when the file contains only valid empty mappings. - -#### `ResolveConfig(repoRoot, modelIdFlag, numCtxFlag string) (*Config, error)` - -Top-level entry point for full configuration resolution. Algorithm: -1. Calls `LoadConfig` to read the YAML file. If the error is a non-existent-file sentinel (`os.IsNotExist(err)`), it swallows and initializes an empty `Config{}`; all other errors propagate wrapped as `"failed to load configuration file:"`. -2. Layers defaults (hard-coded constants) over the loaded config. -3. Layers environment variable overrides using `os.Getenv` for `CODE_REDUCER_MODEL_ID`, `OLLAMA_BASE_URL`, and `OLLAMA_NUM_CTX`. Non-empty values are used; empty strings fall through. Env read errors are swallowed. -4. Applies CLI flags (`modelIdFlag`, `numCtxFlag`). The flag value is parsed via `strconv.Atoi`; only `err == nil && n > 0` is acceptedβ€”invalid or non-positive values cause the prior layer's value to be retained silently. - -Returns a fully resolved `*Config`. Errors are either pass-through (from `LoadConfig`) or wrapped with `"failed to load configuration file:"`. Notable: if YAML is malformed, the parse error wraps and propagates; otherwise resolution always succeeds for missing files. - -### Configuration Persistence - -#### `SaveConfig(cwd string, cfg *Config) error` - -Serializes the config struct back to disk using `os.WriteFile` with `0600` permissions. Error handling: -- Write success β†’ returns `nil`. -- Parse failure (marshal YAML) β†’ wrapped as `"failed to marshal yaml: %w"`. -- File-write failure β†’ wrapped as `"failed to write config file: %w"`. - -Writes directly (no temp-file + rename); a partial write can leave a truncated file on disk. Permission normalization is implicitβ€”existing files with different permissions will fail the write. - -## Utility Functions - -### `MergeAndDeduplicate[T comparable](a, b []T) []T` - -Generic slice merge that deduplicates entries. Merges `b` after `a`; duplicates are removed via a map lookup and never reported as errors. Returns a new slice; the original inputs remain unmodified. The function is safe for concurrent use (no shared mutable state). \ No newline at end of file diff --git a/wiki/modules/internal_engine.md b/wiki/modules/internal_engine.md deleted file mode 100644 index 10aabc0..0000000 --- a/wiki/modules/internal_engine.md +++ /dev/null @@ -1,276 +0,0 @@ -# Module: internal/engine β€” Code Documentation Pipeline Engine - -## Overview - -The `internal/engine` package implements a Map-Reduce pipeline that generates and maintains technical documentation for a Go codebase using Ollama-compatible LLM inference endpoints. The engine produces three artifacts per run: `architecture.md` (global overview), `quickstart.md` (onboarding guide), and per-module API summaries under `modules/`. Execution is gated by a repository-level lock to prevent concurrent runs against the same root, and state persistence across invocations is handled by an on-disk metadata cache. - -**Data flow:** Repository isolation (`security.EnsureGitignoreHasLockfile`) β†’ lock acquisition (`security.AcquireLock`) β†’ LLM client construction from config β†’ pipeline dispatch (`RunInit` or `RunUpdate`) β†’ recursive tree synthesis with bottom-up reduction β†’ standard docs generation β†’ final state persistence. Errors propagate to the caller unless explicitly swallowed and logged via the `Event{Type: "status", Message: ...}` channel. - ---- - -## Pipeline Orchestration - -### Runner Entry Point (`runner.go` β€” `Runner.Run`) - -The public entry point for all pipeline invocations. Responsibilities: repository isolation, concurrency guard acquisition, LLM client construction from config, and mode-gated dispatch to the appropriate pipeline method. - -``` -Run(ctx, repoRoot, mode string, onEvent func(Event)) error - β”œβ”€ ensure .gitignore lockfile entry (best-effort; failure logged as warning) - β”œβ”€ acquire repository-level lock via security.AcquireLock(repoRoot) - β”‚ └─ defer lock.Unlock() β€” released on any return path including errors - β”œβ”€ build *LLMClient from r.cfg.ModelID, BaseURL, NumCtx - β”œβ”€ switch mode: - β”‚ case "init": client.RunInit(ctx, repoRoot, r.cfg, onEvent) - β”‚ case "update": client.RunUpdate(ctx, repoRoot, r.cfg, onEvent) - β”‚ default: return fmt.Errorf("unsupported mode: %s", mode) -``` - -The `cfg *config.Config` field is read-only within this file; all mutations to the runner's configuration originate before construction. The lock returned by `security.AcquireLock(repoRoot)` is a local variable with deferred cleanupβ€”its internal type and mutual exclusion semantics are not observable from this source alone. - -### Orchestrator Pipelines (`orchestrator.go` β€” `RunInit`, `RunUpdate`) - -#### Full Run (`RunInit`) - -Executes an all-or-nothing regeneration regardless of cache state: - -1. **Setup** β€” Load `.gitignore` via `tools.LoadGitignore(repoRoot)`; merge with user ignore list; load metadata cache from disk via `cache.loadMetadataCache()`. Failures logged as warnings, not returned. -2. **Map Phase** β€” Discover code files via `tools.DiscoverCodeFiles(...)`, compute SHA-256 per file via `computeSHA256(repoRoot, f)`, build directory tree via `buildTree(codeFiles)`. Hash computation errors silently skipped in loops; only stored if hash computed successfully. -3. **Mark All Affected** β€” Recursively flag every node as affected (init starts fresh). -4. **Reduce Phase** β€” Call `synthesizeNode` bottom-up on the tree to produce a root summary string and per-module summaries via LLM calls through `c.CallLLM`. -5. **Generate Standard Docs** β€” Produce `architecture.md` and `quickstart.md` from the root summary via two additional `CallLLM` invocations: one for architecture, one for quickstart. Writes via `tools.WriteFileSafely`; errors wrapped with context (e.g., `"failed to write quickstart.md: %w"`). -6. **Update AGENTS.md** β€” Read existing file via `tools.ReadFileSafely(repoRoot, "AGENTS.md")`. If missing, create with guidelines header. If present but does not contain "AI Agent Guidelines", append separator + new content block. Only writes when content actually differs. Errors: if read fails but write succeeds β†’ `"failed to write AGENTS.md: %w"`; if both fail β†’ wrapped errors chain. -7. **Teardown** β€” Persist metadata cache via `saveMetadataCache(repoRoot, docsDir, cache)`. Failure logged as warning, function always returns nil at end regardless of this failure. - -#### Incremental Run (`RunUpdate`) - -Executes only re-generations for directories whose contents actually changed: - -1. **Setup** β€” Load pipeline state from metadata cache (same path as init). -2. **Map Phase** β€” Discover code files, hash each via `computeSHA256`, build tree. -3. **Change Detection** β€” Compare current file hashes against cached hashes in `cache.Files`: - - New file in codebase but absent from cache β†’ `FileChange{Status: "Added"}` - - Existing file with different hash β†’ `FileChange{Status: "Modified"}` - - Cached file no longer in codebase β†’ `FileChange{Status: "Deleted"}` (also removed from cache) -4. **Prune Stale Modules** β€” Remove any module summaries whose directory is no longer active, delete their markdown files on disk via `os.Remove(absModuleFile)` (error assigned to `_`, never logged). -5. **Propagate Affected** β€” Run tree-aware propagation (`propagateAffected`/`determineAffected`) to mark directories affected by changes. Errors from these calls are discarded without capture. -6. **Early Exit** β€” If zero directories are affected β†’ pipeline ends with "up to date" status, no further synthesis occurs. -7. **Conditional Reduce** β€” If root-level change detected or global docs don't exist on disk (`os.Stat(absArch)`/`os.Stat(absQs)`), run full synthesis; otherwise skip and reuse existing `architecture.md`/`quickstart.md`. -8. **Teardown** β€” Save updated cache via `saveMetadataCache(repoRoot, docsDir, cache)`. - -### Directory Tree Construction & Affected Propagation (`tree.go`) - -#### Type Definitions - -| Type | Fields | Purpose | -|------|--------|---------| -| `FileChange` | `Path string`, `Status string` | Models a file operation with path and status (`Added`, `Modified`, `Deleted`). Consumed by `determineAffected`. | -| `DirNode` | `Path string`, `Files []string`, `Children map[string]*DirNode` | Hierarchical tree node holding files at that level and child directories in its subtree. Produced by `buildTree`. | - -#### Functions (all unexported) - -- **`buildTree(codeFiles)`** β€” Converts a flat list of file paths into nested `*DirNode` structure, preserving directory nesting by splitting on `/`. Appends to `DirNode.Files`; populates `DirNode.Children` via map assignment. -- **`propagateAffected(tree, changeset)`** β€” Traverses the tree recursively: if any direct child of the current node is in the changeset, mark it affected. Accumulates affected status upward through ancestors. Returns mutated `affectedDirs`. -- **`determineAffected(tree, changeset)`** β€” Computes module path for each directory via `ToSafeMarkdownFilename`, checks existence on disk via `os.Stat(absModulePath)`, reads metadata cache (`cache.Modules[n.Path] == ""`), marks node affected if any check fails. Only `os.IsNotExist` is acted on; all other errors are silently dropped. - ---- - -## LLM Client Abstraction (`client.go`) - -### Type: `LLMClient` - -```go -type LLMMClient struct { - ModelID string - BaseURL string - NumCtx int - HTTPClient *http.Client // default transport, no custom config (no proxy/redirect/TLS) -} -``` - -Constructed via `NewLLMClient(modelID, baseURL, numCtx)` with a 10-minute HTTP timeout. Fields set once; never reassigned within any method. - -### Methods on `*LLMClient` - -#### `CallLLM(ctx context.Context, systemPrompt string, messages []Message, jsonFormat bool) (string, error)` - -Full lifecycle: prepends system prompt as first message with role `"system"` β†’ serializes into Ollama `/api/chat` JSON payload with model ID and context window size β†’ non-streaming HTTP POST β†’ response body fully read on 200 OK or truncated at 1 KB via `io.LimitReader(resp.Body, 1024)` on non-OK status. Returns only assistant content string; returns empty string on failure. - -**Error behavior:** -| Failure path | Error type | Wrapping strategy | -|---|---|---| -| JSON marshal fails in request prep | `fmt.Errorf("failed to marshal request: %w", err)` | Wrapped with `%w` | -| `http.NewRequestWithContext` fails | raw error from `NewRequestWithContext` | **Not wrapped** β€” passed through as-is | -| HTTP Do (network-level failure) | raw error from `c.HTTPClient.Do(req)` | **Not wrapped** | -| Body read fails on 200 OK path | raw error from `io.ReadAll` | **Not wrapped** | -| JSON unmarshal fails on response | `fmt.Errorf("failed to parse response: %w", err)` | Wrapped with `%w` | -| Non-OK status (body truncated at 1 KB) | `fmt.Errorf("ollama api error: status %d, response: %s", ...)` | New error constructed; body discarded if read fails | - -#### `GetBaseSystemPrompt() string` - -Returns the fixed base role definition and defensive guidelines inherited by every LLM call regardless of task type. - -#### `GetDefaultSystemPrompt(command string) string` - -Appends command-specific instructions based on the command string: two named variants exist (`module_synthesis`, `architecture`); any other value receives only the base layer. - ---- - -## Response Parsing Layer (`json_parser.go`) - -### Functions (all unexported except `StripOuterMarkdownFence`, `CleanJSONResponse`, `UnmarshalJSONResponse`) - -Internal helper `extractBalancedJSON` is lowercase and excluded from public surface area. - -#### Business Rules - -1. Markdown code fences (` ``` `) surrounding JSON must be stripped before parsing β€” handled by `StripOuterMarkdownFence`. -2. Balanced bracket matching required β€” strings containing `{`, `[`, `}`, `]` literals (including escaped characters like `\n`) must not break extraction logic β€” handled by internal rune-level scanner with escape state tracking inside string literals. -3. When multiple balanced JSON candidates exist in a single response, each must be attempted individually until one succeeds, or the entire trimmed string is tried as a final fallback β€” handled by `CleanJSONResponse`/`UnmarshalJSONResponse`. - -#### Functions - -| Function | Behavior | Error surface | Swallowed? | -|---|---|---|---| -| `StripOuterMarkdownFence(s string) string` | Strips surrounding markdown fences; falls back to trimmed input if none match | None β€” returns string only | N/A (no error return) | -| `CleanJSONResponse(s string) string` | Iterates stripped content, calls `extractBalancedJSON`, returns first successful candidate or original trimmed input unchanged | None β€” returns string only | Yes β€” unbalanced JSON inside response is swallowed silently | -| `UnmarshalJSONResponse(s string, target interface{}) error` | Multi-layered attempt: (1) collect balanced candidates, try each into target; keep last error in `lastErr`; (2) if any succeeded, return nil; otherwise try entire trimmed string; (3) terminal errors β€” `"failed to unmarshal JSON candidates: %w"` or `"no valid JSON found in response"` | Explicit (`error`) | Partial parse failures collected and re-emitted via `%w` on first path; "no valid JSON" covers second path | - ---- - -## Synthesis Engine (`synthesize.go`, `chunking.go`) - -### Context-Aware Chunking (`chunking.go` β€” all unexported) - -The system processes extracted code/fact items in batches constrained by LLM context limits (`NumCtx * 3`). Three distinct processing modes exist: - -#### Module Synthesis (`reduceInChunks`) - -Synthesizes architectural-level nodes into architecture descriptions using a `"module_synthesis"` system prompt. Each batch represents evidence contributing to understanding the module's design intent. Response passes through `StripOuterMarkdownFence`. Errors wrapped with `"LLM error during synthesis: %w"`. - -#### File Fact Consolidation (`reduceFileFacts`) - -Consolidates extracted facts about a specific step within a file with deduplication and merging. System prompt built locally by concatenating `c.GetBaseSystemPrompt()` + fixed role string. Response also passes through `StripOuterMarkdownFence`. Errors wrapped with `"LLM error during file fact consolidation: %w"`. - -#### Recursive Convergence (`reduceItems`) - -Core business rule: reduce items in batches β†’ collect intermediate results β†’ recursively apply reduction until everything converges into a single output string. Multi-pass approach ensures information is progressively synthesized rather than lost across multiple LLM calls. Termination guaranteed when all data fits within one batch (short-circuits at top with `len(batches) == 1`). Recursion terminates because: -1. Single-batch case short-circuits at top (`len(batches) == 1`) -2. Recursive step processes each batch through itself again with same `maxChars` - -**Graceful Truncation Policy:** When content exceeds the context window, single oversized items get truncated with `\n...[truncated]` marker appended; when multiple items exceed limits collectively, each gets capped at `maxChars / len(items)` characters before being processed as one batch. Infinite-loop guard: if batch size equals item count for all items (can't merge), falls back to per-item truncation using `allowedPerItem = maxChars / len(items)`. Still silent β€” no error returned. - -#### Overlapping Chunk Support (`chunkTextWithOverlap`) - -Utility for splitting raw text into overlapping chunks to maintain context continuity between adjacent segmentsβ€”useful when feeding source material directly into LLMs without pre-processing through the chunking logic. No I/O. Silent swallow: `maxRunes <= 0` β†’ returns full text as single chunk; `overlapRunes >= maxRunes` β†’ clamps to half without warning; final partial chunk reaching end of runes β†’ breaks loop with no error or truncation marker (unlike `reduceItems`). - -### Hierarchical Module Synthesis (`synthesize.go` β€” `synthesizeNode`, all unexported) - -Performs recursive synthesis of directory structures into hierarchical summariesβ€”transforming raw source code and subdirectory metadata into consolidated documentation at each level of a module tree. - -**Algorithmic Flow:** -1. **Context Check β†’ Cache Reuse Gate**: If current node not in `affectedDirs` AND cached module summary exists, return immediately without processing. Short-circuits unaffected directories. -2. **Recursive Child Synthesis (Bottom-Up)**: Sort children alphabetically, recursively call `synthesizeNode` on each child directory; results collected into map keyed by child name. -3. **File Processing Loop**: For each file in current node: hash-first cache lookup (if precalculated hash exists and matches cached SHA256 β†’ reuse cached facts without reading disk); read + hash fallback otherwise (read via `tools.ReadFileSafely`, compute SHA256, update cache entry if already existed with that hash); extraction pipeline on cache miss (chunk content using dynamic limit derived from `c.NumCtx Γ— 4 characters Γ— 0.75` with minimum of 512 tokens and max overlap cap; for each chunk iterate over every step in `cfg.ExtractionSteps`, sending to LLM with system prompt built from base + step-specific prompt; consolidate all chunks for that step via `reduceFileFacts`; append consolidated result under step's name heading). -4. **Component Assembly**: After processing files, append any non-empty child summaries to components list. If no components exist at all (no files AND no children), clear module cache and return empty string. -5. **Final Synthesis & Persistence**: Pass assembled components to `reduceInChunks` for directory path; store result in both memory cache (`cache.Modules[node.Path]`) and on disk at `//modules/` via `tools.WriteFileSafely`; return final summary string. - -**Domain concepts:** Hierarchical module synthesis (bottom-up: files contribute facts, subdirectories contribute their own summaries, parent combines both); context-aware truncation (file content limits scale with `c.NumCtx`, reserving 75% of tokens for file data and 25% for prompts/output); two-level caching (files cache individual facts keyed by SHA256, modules cache full synthesized summaries keyed by path; reuse only when node not affected); multi-step extraction pipeline (each file passes through configurable sequence of LLM steps, each producing independent fact set consolidated before combining with other files' results). - -**Error handling in `synthesizeNode`:** -| Source | Pattern | Propagation | -|---|---|---| -| Context cancel | Direct return at top and inside per-file loop (`if err := ctx.Err(); err != nil { return "", err }`) | Upward, no wrapping | -| LLM call failure | Wrapped: `"LLM error extracting %s for %s: %w"` with step name + file path | Upward | -| File read failure | Logged via `logEvent("status", ...)`, skipped silently; no hash/facts/anything added to components | None (swallowed) | -| Module write failure | Wrapped: `"failed to write module documentation for %s: %w"` with dir path | Upward | -| Recursive child error | Propagated as-is (`return "", err`) | Upward | -| Empty components | Returns `("", nil)`, clears `cache.Modules[node.Path] = ""` | None (clean return) | - ---- - -## Filesystem State Management (`cache.go`) - -### Type: `MetadataCache` - -```go -type MetadataCache struct { - Files map[string]FileCacheEntry // virtual path β†’ cached SHA256 + facts - Modules map[string]string // module identifier β†’ summary string -} -``` - -Both maps initialized with `make()` when nil; populated during load/save lifecycle; serialized to disk. No in-memory locking applied within this file. - -### Functions (all unexported except `IsInitialized`) - -#### Core Concept: Filesystem-Based State Cache - -Persists an in-memory cache to disk so expensive computations can be avoided across runs. Stored as `docsDir/.metadata.json`, holds two pieces of state: -- **File-level metadata** (`Files map`): maps virtual paths (filenames) to cached SHA256 hashes plus associated "facts" strings. -- **Module-level metadata** (`Modules map`): maps module identifiers to string values. - -#### Functions - -| Function | Direction | Path constructed | Notes | -|---|---|---|---| -| `loadMetadataCache()` | Read | `docsDir/.metadata.json` | Attempts to read JSON; if absent, returns freshly initialized empty cache (not error); if present but unparseable, errors. Null maps normalized to empty on load. | -| `IsInitialized(repoRoot, docsDir string) bool` | Read (existence probe) | `docsDir/.metadata.json` | Probes for existence of `.metadata.json`. Successful read β†’ true; any non-nil error β†’ false. N/A for missing file (returns false). | -| `saveMetadataCache()` | Write | `docsDir/.metadata.json` | Serializes struct back to JSON with indentation, writes atomically via `tools.WriteFileSafely`. Returns raw errors from marshal/write unchanged. | -| `computeSHA256(virtualPath)` | Read (virtual path) | caller-supplied virtual path | Reads file by virtual path, runs SHA-256 on content, returns hex-encoded digest. No disk I/O within this file; pure in-memory (`crypto/sha256`). | - -**Error handling patterns:** -| Function | On missing cache file | On generic read/write error | On JSON parse error | -|---|---|---|---| -| `loadMetadataCache` | Returns empty initialized `*MetadataCache`, nil error | Wraps with `"failed to read metadata cache: %w"` | Wraps with `"failed to unmarshal metadata cache: %w"` | -| `IsInitialized` | N/A (returns false) | N/A β€” any non-nil error β†’ false | β€” | -| `saveMetadataCache` | β€” | Returns raw error from `tools.WriteFileSafely` | Returns raw error from `json.MarshalIndent` | -| `computeSHA256` | β€” | Returns raw error from `tools.ReadFileSafely` | N/A (no parsing) | - ---- - -## Utility Functions (`utils.go`) - -### Function: `ToSafeMarkdownFilename(modulePath string) string` - -Pure string transformation with no business logic or complex algorithmic flow. Maps a module path to a safe filename for markdown documentation. - -**Conversion pipeline:** -1. Replace every `/` character in input `modulePath` with `_`. -2. If result is empty or equals `"."`, substitute with `"root"`. -3. Append `.md` and return. - -**Domain concept:** Module-to-document routing β€” given a module identifier, produce corresponding markdown file path. Assumes caller already owns consistent `modulePath` convention; this utility only adapts that path for filesystem-safe filenames. - ---- - -## Concurrency & State Summary - -### Mutable State Across Files - -| Variable | File | Mutated By | Description | -|---|---|---|---| -| `MetadataCache.Files` | `cache.go` | load/save lifecycle | Map[string]FileCacheEntry β€” populated during load/save; serialized to disk | -| `MetadataCache.Modules` | `cache.go` | load/save lifecycle | Map[string]string β€” same pattern as Files | -| `DirNode.Files` | `tree.go` | buildTree only | Appended in tree construction, read-only otherwise | -| `DirNode.Children` | `tree.go` | buildTree only | Populated via map assignment during tree construction | -| `affectedDirs` (map[string]bool) | `orchestrator.go`, `tree.go` | Recursive closures: markAllAffected, collectDirs, determineAffected, propagateAffected | Populated by recursive traversal of tree nodes; passed to synthesizeNode. Local scope only in each invocation. | -| `cache.Files` / `cache.Modules` (local) | `orchestrator.go`, `synthesize.go` | synthesizeNode β†’ updateCacheFromHashes, buildModuleMap, etc. | Populated in RunInit; modified during synthesis in RunUpdate. Local to function invocations; no shared mutable state across goroutines visible here. | -| `cfg *config.Config` (field on Runner) | `runner.go` | Set once in NewRunner; not mutated within this file | Read through methods delegated to other packages only. External mutation possible before construction, but no writes occur here. | - -### Concurrency Mechanisms: None Detected - -No `sync.Mutex`, channels (`chan`), atomic operations, or other synchronization primitives appear in any of these files. The recursion in `synthesizeNode` is itself a concurrency primitive (caller may invoke from many goroutines), so map writes to `cache.Files` and `cache.Modules` are unprotected against concurrent access if called with multiple goroutinesβ€”but no such protection exists within this codebase's visible scope. - -### External Communication Summary - -| File | I/O Category | Targets | -|---|---|---| -| `client.go` | Network (HTTP POST) | Ollama-compatible endpoint at `{BaseURL}/api/chat`, 10-minute timeout, non-streaming | -| `json_parser.go` | None | Pure in-memory string/byte manipulation; only imports from standard library (`encoding/json`, `fmt`, `regexp`, `strings`) | -| `cache.go` | Disk (read/write) | `docsDir/.metadata.json` via `tools.ReadFileSafely` / `tools.WriteFileSafely` | -| `orchestrator.go` | Network + Disk | LLM service calls, local filesystem under `repoRoot` (md files, cache, dir tree), `.gitignore` read/write | -| `runner.go` | Disk write (lockfile) | `.gitignore` modification for lockfile entry; lock file creation/update via `security.AcquireLock`; defer cleanup on any return path | -| `synthesize.go` | Network + Disk | LLM calls per chunk per extraction step, file reads via `tools.ReadFileSafely`, final writes via `tools.WriteFileSafely` with error wrapping | -| `tree.go` | Disk read (one op) | `os.Stat(absModulePath)` for module existence check; all other operations are pure tree construction/traversal | -| `utils.go` | None | Pure string transformation, no external I/O whatsoever | \ No newline at end of file diff --git a/wiki/modules/internal_security.md b/wiki/modules/internal_security.md deleted file mode 100644 index eae9168..0000000 --- a/wiki/modules/internal_security.md +++ /dev/null @@ -1,137 +0,0 @@ -# `internal/security` β€” Package Documentation - -## Module Responsibility & Data Flow - -The `security` package encapsulates three orthogonal concerns for code-reducer processes: **sandbox enforcement** (preventing path traversal beyond the repository boundary), **process-level mutual exclusion** via file-based locks, and **operational hygiene** (keeping lock state out of version control). No networking, database access, or external process communication occurs within this package. - -All functions are pure-Go with no global mutable state. Every observable side effect is local to a single `*SimpleLock` receiver instance or the calling goroutine's stack. Errors follow a strict wrapping convention: security-critical violations and actionable sentinel messages return unwrapped plain strings; standard-library errors use `%w`; best-effort cleanup paths swallow non-fatal I/O failures silently. - -**Data flow** proceeds in two independent tracks that share no state except through the repository root path string: - -1. **Path resolution track:** `SafeResolve` β†’ called by `AcquireLock` to canonicalize the lock file location before attempting acquisition. -2. **Lock lifecycle track:** `AcquireLock` β†’ returns a `*SimpleLock` receiver β†’ caller uses it for work β†’ calls `Unlock()` for release β†’ `EnsureGitignoreHasLockfile` runs during cleanup to append the lock filename to `.gitignore`. - ---- - -## Public API Surface - -### Types - -#### `SimpleLock` - -File-based process lock with internal mutex protection. The struct is constructed by `AcquireLock`; once acquired, `lockPath` and `file` are set exactly once under mutex scope, then only modified during `Unlock()`. - -| Field | Type | Semantics | -|-------|------|-----------| -| `lockPath` | `string` | Absolute path to the `.code-reducer.lock` file. Set in constructor; zero-valued if acquisition failed or was never called. | -| `file` | `*os.File` | Opened lock file handle. Acquired with `O_CREATE\|O_EXCL`; set to `nil` inside `Unlock()`. | -| `mu` | `sync.Mutex` | Per-instance mutex protecting `closed` and the pointer-clear of `file` during unlock. | -| `closed` | `bool` | Transition flag: `false` while locked, flips to `true` once `Unlock()` completes its write operations (under mutex). | - -#### Functions - -##### `SafeResolve(repoRoot, inputPath string) (string, error)` - -Computes the absolute path of an input relative to a repository root. The resolution is symlink-aware: ancestor components are evaluated through their physical targets via `filepath.EvalSymlinks` before being walked upward until one succeeds. The final result must lie strictly inside the resolved rootβ€”any escape beyond it returns an unwrapped string error indicating a security violation. - -##### `AcquireLock(repoRoot string) (*SimpleLock, error)` - -Resolves the lock file path through `SafeResolve`, then attempts atomic creation with `O_CREATE\|O_EXCL`. On success, writes the process PID to the newly created file and returns a populated `*SimpleLock`. If the file already exists (anewer process holds it), returns an unwrapped sentinel string describing that another process has acquired the lock. - -##### `EnsureGitignoreHasLockfile(repoRoot string) error` - -Reads `.gitignore`; if present, appends `# Code-Reducer Lockfile\n.code-reducer.lock\n` only when not already contained in the file. If `.gitignore` does not exist, creates it with append semantics (`O_APPEND\|O_CREATE\|O_WRONLY`). Errors from reading or writing are wrapped; non-existence of `.gitignore` is treated as expected and returned as `nil`. - -#### Methods - -##### `(l *SimpleLock).Unlock() error` - -Idempotent release: acquires the internal mutex, closes the file handle (discarding any close-error), removes the lock file from disk via best-effort `os.Remove`, sets `closed = true`, then releases the mutex. A second call is safe and returns `nil`. Swallowed errors for cleanup operations are discarded; removal failures that occur after a successful close surface as wrapped errors only in the specific narrow case where both close-succeeded-and-remove-failedβ€”but this path effectively swallows non-`IsNotExist` remove errors in practice. - ---- - -## Business Rules & Domain Concepts - -### Repository Boundary Enforcement - -All internal paths must resolve strictly within `repoRoot`. Any resolution that escapes beyond it returns a standalone string error: `"security violation: path traversal detected: %q"`, never wrapped. This is the one unwrapped class of non-sentinel errors in the packageβ€”security-critical, so callers detect it via exact match or `strings.Contains` rather than type assertion. - -### Symlink-Aware Path Resolution - -Ancestors are evaluated through their physical targets before path reconstruction. This prevents traversal attacks that use symlinked directories to bypass the sandbox while preserving logical input structure for legitimate uses. Implementation: `filepath.EvalSymlinks(absRoot)` resolves the root once; subsequent ancestor walks use `os.Lstat` with a fallback to `EvalSymlinks` on each component until one exists, then rebuild upward from that point. - -### Process-Level File Locking - -The lock file is `.code-reducer.lock`. Acquisition uses `O_EXCL` for atomic creationβ€”no two processes can hold the lock simultaneously on the same filesystem. On success, the PID of the acquiring process is written as a single-line string (`fmt.Sprintf("%d\n", os.Getpid())`). The lockfile content therefore serves as both a record and a mechanism for stale detection: if another process exits without releasing its lock, a new acquisition attempt detects the existing file and reports an actionable error instructing manual cleanup. - -### Stale Lock Detection - -Intentional behaviorβ€”the system does not attempt to detect or kill the holding process. The caller receives an unwrapped string describing that the lock is held by another process and must clean up manually. This keeps the package non-invasive; it never interacts with external processes. - -### Automated Gitignore Integration - -The lockfile path is appended to `.gitignore` during unlock so it does not enter version control. Idempotent: if a line containing `code-reducer.lock` already exists, no duplicate entry is written. The lock filename itself is the only thing tracked; comment prefix provides human readability without affecting git behavior. - ---- - -## Mutable State & Concurrency Analysis - -### Per-Instance State - -| Field | Mutated By | Notes | -|-------|-----------|--------| -| `lockPath` | Constructor (`AcquireLock`) | Set exactly once. Zero value if acquisition failed or never called. Not mutated after construction. | -| `file` (*os.File) | Constructor, then `Unlock()` | Pointer cleared inside the mutex during unlock so subsequent reads yield nil. | -| `mu` (sync.Mutex) | All methods on receiver | Acquired/released by every method; not modified as a value. | -| `closed` (bool) | `Unlock()` only | Flips from `false` β†’ `true` under mutex. Once written, stays true for the lifetime of the receiver. | - -### Concurrency Protection - -The `sync.Mutex` protects all writes to `l.closed` and the pointer-clear of `l.file` inside `Unlock()`. No other concurrent access paths touch these fields in this fileβ€”`AcquireLock` constructs a new instance, so no shared writer exists for an already-acquired lock. The mutex serializes unlock calls on a single receiver only; it is not a global contention point across processes. - ---- - -## Side Effects & I/O Communication - -### Filesystem Operations by Function - -| Function | Reads | Writes | Deletes/Atomic | Notes | -|----------|-------|--------|----------------|--------| -| `SafeResolve` | `os.Lstat`, `filepath.EvalSymlinks` (metadata only) | None | None | Walks ancestors upward until one exists; no data read/written. | -| `AcquireLock` | None | `O_CREATE\|O_EXCL` + PID bytes via `fmt.Sprintf("%d\n", os.Getpid())` | None | Atomic creation on success; cleans up file handle and removes partial state if PID write fails before returning error. | -| `SimpleLock.Unlock()` | None | `l.file.Close()` (flushes buffer) | `os.Remove(l.lockPath)` β€” best-effort, non-blocking | Idempotent. Swallowed close errors; remove failures are discarded unless they occur after a successful close and removal itself fails. | -| `EnsureGitignoreHasLockfile` | `os.ReadFile(gitignorePath)` | Append-mode open + write string | None | Creates `.gitignore` if absent (append semantics). Appends only when lock filename is not already present. | - -### Network / External Process - -None. The package performs no HTTP, gRPC, socket, or IPC operations. No external process interaction beyond reading/writing files within the repository boundary. - ---- - -## Error Handling Patterns - -### Sentinel Errors (Unwrapped) - -| Condition | Returned Value | Rationale | -|-----------|---------------|-----------| -| Lock held by another process | `"lock at %s is already held by another process..."` | Actionable message; caller needs exact match or substring search. | -| Path traversal detected | `"security violation: path traversal detected: %q"` | Security-critical; never wrapped for unambiguous detection. | - -### Wrapped Errors (Standard Library) - -| Condition | Wrapping Strategy | Notes | -|-----------|------------------|--------| -| `filepath.Abs` fails | `%w` | Root is not absoluteβ€”caller should handle gracefully. | -| `filepath.EvalSymlinks` on root fails | `%w` | Symlink resolution failure for the repository boundary itself. | -| `os.Lstat` returns non-nil, non-IsNotExist error | `%w` | Unexpected stat error on an ancestor component. | -| `EvalSymlinks` on ancestor fails | `%w` | Ancestor symlink resolution failure during upward walk. | -| `.gitignore` read (not IsNotExist) | `%w` | Read of existing gitignore encounters unexpected I/O error. | -| Append/open for `.gitignore` fails | `%w` | File open or write failure for the ignore file. | - -### Swallowed Errors (Best-Effort Cleanup) - -If `l.file.Close()` inside `Unlock()` returns an error, it is replaced with `nil` and cleanup proceeds unconditionally. If `os.Remove(l.lockPath)` fails after a successful close, the remove-error is discarded unless both close-succeeded-and-remove-failedβ€”but this narrow path effectively swallows non-`IsNotExist` errors in practice. The lockfile removal happens regardless; any failure that isn't `IsNotExist` is preserved in a local variable but only surfaces when close succeeded and remove failedβ€”which is treated as swallowed behavior: best-effort cleanup. - -### Lock Acquisition Failure Cleanup - -In `AcquireLock`, if writing the PID to the newly created file fails, the function calls `f.Close()` then `os.Remove(lockPath)` before returning an error. The file handle and lockfile are cleaned up so no stale partial state remains on disk; the returned error is wrapped with `%w` for context. \ No newline at end of file diff --git a/wiki/modules/internal_tools.md b/wiki/modules/internal_tools.md deleted file mode 100644 index dacd13d..0000000 --- a/wiki/modules/internal_tools.md +++ /dev/null @@ -1,110 +0,0 @@ -# `internal/tools` β€” File System & Git Operations Module - -## Module Responsibility - -This module provides two complementary interfaces for repository-aware operations: safe file I/O with TOCTOU race mitigation, and Git process abstraction for executing commands within the working tree. Both modules operate on a single root path (`repoRoot`) without shared mutable state, and neither uses concurrency primitives β€” all functions are goroutine-safe by construction. - ---- - -## `file_tools.go` β€” Safe File I/O & Discovery - -### Core Functions - -| Function | Input | Output | -|---|---|---| -| [`ReadFileSafely`](file:///path/to/file_tools.go#L15-L64) | `repoRoot`, `virtualPath` `string` | `[]byte`, `error` | -| [`WriteFileSafely`](file:///path/to/file_tools.go#L70-L132) | `repoRoot`, `virtualPath` `string`; `content` `[]byte` | `error` | -| [`LoadGitignore`](file:///path/to/file_tools.go#L138-L164) | `repoRoot` `string` | `[]string`, `error` | -| [`ShouldIgnoreFile`](file:///path/to/file_tools.go#L170-L236) | `repoRoot`, `relPath` `string`; `gitIgnore` `*ignore.GitIgnore`; `ignoredExtensions` `[]string` | `bool` | -| [`DiscoverCodeFiles`](file:///path/to/file_tools.go#L242-L291) | `repoRoot` `string`; `ignores`, `ignoredExtensions` `[]string` | `[]string`, `error` | -| [`IsBinaryFile`](file:///path/to/file_tools.go#L297-L315) | `path` `string` | `bool` | - -### TOCTOU Race Mitigation Pattern - -Both `ReadFileSafely` and `WriteFileSafely` implement a Time-Of-Check-Time-Of-Use safety pattern: - -1. Resolve the virtual path to an absolute safe path via `SafeResolve` -2. Open the file descriptor first -3. Perform `lstat` on the resolved path (no symlink following) -4. Perform `fstat` on the open file descriptor (follows symlinks) -5. Verify that `lstat` and `fstat` reference the same inode β€” mismatch indicates a symlink race between opening and checking - -Both operations reject any symlink detection as a security violation. On success, `ReadFileSafely` reads all content via buffered I/O; on success, `WriteFileSafely` truncates to zero first (safe only because symlinks were ruled out), then writes the provided content. - -### Error Handling Contract - -| Function | Wrapped Errors (`%w`) | Swallowed | Sentinel Unwrapped | -|---|---|---|---| -| `ReadFileSafely` | Yes, for every I/O error | No | Symlink race check | -| `WriteFileSafely` | Yes, for every I/O error | No | Symlink race check | -| `LoadGitignore` | No | Missing file β†’ `(nil, nil)` | None | -| `ShouldIgnoreFile` | No | `SafeResolve` err β†’ returns `true` | None | -| `DiscoverCodeFiles` | No | Extensive β€” walk callback errors swallowed at each node | None | -| `IsBinaryFile` | No | All open/read errors β†’ `false` | None | - -### File Discovery Algorithm (Multi-Layer Ignore Rules) - -A file is considered "ignored" if it matches **any** of these conditions: - -- Matches a compiled gitignore pattern list -- Has a path component starting with `.` or ending in `.egg-info` -- Has an extension matching the user-provided ignored extensions list (case-insensitive, supports both `.` prefix and suffix forms) -- Is detected as binary by reading up to 1024 bytes and scanning for null byte (`\x00`) - -If any single check returns true, the file is excluded. This is a **union** of ignore conditions. - -The primary business rule is to recursively walk the repository root and discover high-signal source code files while filtering out noise: build artifacts, dependency directories, output files, dot-prefixed hidden paths, `.egg-info` markers, binary files, and any user-defined ignore patterns (via gitignore compilation). - -### Binary Detection Rule - -A file is considered binary if its first 1024 bytes contain a null byte (`\x00`). Files that cannot be opened are treated as non-binary (returns false), avoiding the loading of entire files into memory for classification. - ---- - -## `git_tools.go` β€” Git Process Abstraction - -### Core Functions - -| Function | Input | Output | -|---|---|---| -| [`RunGit`](file:///path/to/git_tools.go#L15-L64) | `repoRoot string`, variadic `args ...string` | `(string, error)` | -| [`VerifyGitRepo`](file:///path/to/git_tools.go#L70-L89) | `repoRoot string` | `error` | - -### Execution Model - -Both functions spawn the `git` binary via `exec.Command("git", ...)` with `--no-pager`, restricting execution to the provided root directory. Both capture stdout and stderr into buffers synchronously. No standard library packages beyond `bytes`, `fmt`, `os/exec`, `strings` are imported. - -### Error Handling Contract - -| Function | Failure Path | Behavior | -|---|---|---| -| `RunGit` | Exit code β‰  0 | Wraps error: `git command failed: %v, stderr: %s`. Returns `(string, error)` with stdout and wrapped error. No panic. | -| `RunGit` | Exit code 0 | Returns `(trimmedOut, nil)`. Stdout is whitespace-trimmed; stderr is discarded on success. | -| `VerifyGitRepo` | `RunGit(...)` returns error | Wraps it: `not a git repository (or any of the parent directories)`. The original stderr content is lost in this wrapper. Returns only the wrapped sentinel-like error. No panic. | - -### Repository Integrity Check - -A directory is considered a valid Git working tree only if `git rev-parse --is-inside-work-tree` succeeds; any failure indicates the path is outside a Git working tree or not a git repository at all. The verification phase delegates to the execution phase with specific arguments, interpreting success as confirmation of repository validity and failure as an assertion that the directory is not a Git repository. - ---- - -## Data Flow Summary - -``` -DiscoverCodeFiles(repoRoot, ignores, ignoredExtensions) -β”œβ”€β”€ Compile gitignore from ignore lines β†’ gitIgnore object (in-memory, no I/O) -└── filepath.WalkDir(repoRoot): - β”œβ”€β”€ Skip repo root itself (first entry) - β”œβ”€β”€ If directory: - β”‚ β”œβ”€β”€ Check name against dot-prefix / .egg-info / gitIgnore - β”‚ └── If any match β†’ Skip entire subtree - └── If file: - └── ShouldIgnoreFile check: - β”œβ”€β”€ gitignore matches? β†’ ignore - β”œβ”€β”€ path component starts with "." or ends ".egg-info"? β†’ ignore - β”œβ”€β”€ extension in ignoredExtensions list? β†’ ignore - β”œβ”€β”€ binary? β†’ ignore - └── otherwise β†’ add to result list - -Result: flat list of relative paths representing source code files that pass all safety and filtering rules. -``` \ No newline at end of file diff --git a/wiki/modules/root.md b/wiki/modules/root.md deleted file mode 100644 index 1dab364..0000000 --- a/wiki/modules/root.md +++ /dev/null @@ -1,307 +0,0 @@ -# Code-Reducer β€” Architecture Documentation - ---- - -## Module Responsibility & Data Flow - -Code-Reducer is a terminal-based documentation-generation agent that produces and maintains technical wikis from Go source code by invoking an Ollama-compatible LLM endpoint. The module implements three phases: **initialization** (repository scan + wiki scaffolding), **interactive setup** (configuration prompts β†’ `.code-reducer.yaml` persistence), and **incremental update** (change detection β†’ selective regeneration). All pipeline execution routes through a single `cmd.RootCmd.Execute()` entry point, with subcommand handlers delegating to an external executor. - ---- - -## Package: cmd β€” CLI Command Wiring & Lifecycle Management - -### Root Command Initialization (`root.go`) - -The root command registers two persistent flags via `PersistentFlags`: `--model-id` (bound to package-level `modelIdFlag`) and `--num-ctx` (bound to package-level `numCtxFlag`). On invocation, the following sequence executes: - -1. **Environment validation** β€” `os.Getwd()` resolves the current working directory; failure is wrapped as `"failed to get current working directory: %w"` and returned immediately. -2. **Configuration resolution** β€” merges CLI flags, environment variables, and an optional persistent config file loaded via `internal/config`. Resolution details are delegated outside this package boundary. -3. **Implicit setup on first run** β€” if the persistent config file does not exist *and* stdin is a terminal (`isatty.IsTerminal(os.Stdin.Fd())` / `isatty.IsCygwinTerminal(os.Stdin.Fd())`), an automatic initial configuration step triggers without requiring user intervention. On non-TTY (CI / piping), this step is skipped and the caller must configure manually via `setup`. -4. **Mode-gated lifecycle checks** β€” enforced only within the runner/engine layer; no validation of project state occurs at registration time. The two modes (`init`, `update`) are mutually exclusive on an already-initialized project; enforcement lives in the external executor. -5. **Graceful shutdown** β€” installs SIGINT / SIGTERM handlers via `signal.NotifyContext`. A deferred `stop()` cancels the context; any in-flight work that respects the cancelled context will exit. No error-return path exists for signal handling within this file. -6. **Execution loop** β€” delegates to `engine.NewRunner.Run()` with the resolved config and current directory. The runner emits typed events (`status`, `error`, default). - -Cobra output suppression: `RootCmd` has `SilenceErrors: true` and `SilenceUsage: true`. Errors that would otherwise be printed by cobra's default handler are suppressed β€” callers must handle errors explicitly. No panic calls observed; all failure paths return errors propagated to cobra. - -### Interactive Configuration Setup Flow (`setup.go`) - -The `RunSetupFlow(repoRoot)` function generates a `.code-reducer.yaml` configuration file through an interactive prompt loop: - -1. **Resolve existing state** β€” attempt to load any previously saved config from the current working directory via `config.LoadConfig(repoRoot)`. If none exists, all prompts default to built-in constants (Ollama defaults for model ID, base URL, context size; `"wiki"` for docs directory). -2. **Prompt user iteratively** for each setting: LLM Model ID β†’ string, Ollama Base URL β†’ string, Ollama Context Size β†’ integer (parsed via `strconv.Atoi`), custom directories/files to ignore β†’ comma-separated list, file extensions to ignore β†’ comma-separated list, documentation directory β†’ string. -3. **Preserve existing values on empty input** β€” if the user presses Enter without typing anything, the previously loaded (or default) value is kept. -4. **Support reset semantics** β€” for list-type inputs, the literal strings `"clear"` or `"none"` wipe the list entirely; otherwise comma-splitting produces the new list. -5. **Error swallowing on read failures** β€” if stdin reading fails during any prompt (`bufio.NewReader(os.Stdin)` + `reader.ReadString('\n')`), the error is logged to stderr and execution continues using existing values (no rollback, no restart). -6. **Persist final config** β€” after all prompts complete, a single `config.Config` struct is constructed from user input + preserved state and written to disk via `config.SaveConfig`. The success message references a `.yaml` file path defined in the config package. - -Non-destructive defaults: existing configuration is always preferred over built-in constants; new values only replace what the user explicitly provided (or left as default). Idempotent re-run: running `setup` again when a config already exists produces an incremental update rather than a full overwrite. No retry logic β€” `SaveConfig` is called once; if it fails, the entire flow aborts. - -### Update Subcommand Registration (`update.go`) - -This file exports no public functions, structs, or interfaces. The handler registers the CLI entry point via `init()` (package-level) and delegates to `executeCommand("update")`, which resolves the actual update logic outside this package boundary. All substantive logic is deferred to the external function; this file serves only as registration/wiring within the cobra command hierarchy. - -### Init Subcommand Registration (`init.go`) - -This file exports no public functions, structs, or interfaces. The handler registers the CLI entry point via `init()` (package-level) and delegates to `executeCommand("init")`, which performs the repository scan and wiki page generation outside this package boundary. All substantive logic is deferred to the external function; this file serves only as registration/wiring within the cobra command hierarchy. - ---- - -## Package: internal/config β€” Multi-Layer Configuration Resolution - -### Responsibility - -Implements a four-tier precedence pipeline for resolving all configurable values. Each layer overwrites prior values when present; missing layers are transparently skipped. Three properties are resolved independently (`ModelID`, `OllamaBaseURL`, `OllamaNumCtx`). The package also structures the analysis pipeline into named **extraction steps** (`ExtractionStep`), each carrying a distinct prompt template. All non-structural state (resolved config, local variables) is ephemeral β€” scoped to function calls and returned as values. No package-level mutable state survives beyond initialization. Concurrency primitives are absent; the package is single-threaded in design. - -### Types - -#### `Config` - -```go -type Config struct { - ModelID string // resolved model identifier (last-wins: default β†’ YAML β†’ env β†’ flag) - OllamaBaseURL string // resolved base URL (default β†’ YAML β†’ env; CLI flag omitted) - OllamaNumCtx int // resolved context size, validated positive (default β†’ YAML β†’ env β†’ flag) - DocsDir string // path to the documents directory under analysis - ExtractionSteps []ExtractionStep // ordered list of extraction phases - Ignore []string // absolute paths excluded from traversal - IgnoreExtensions []string // file extensions excluded from traversal -} -``` - -#### `ExtractionStep` - -```go -type ExtractionStep struct { - Name string // step identifier (e.g., "api-signatures", "business-logic") - Prompt string // prompt template applied to this phase during analysis -} -``` - -Instances are held in a package-level slice (`DefaultExtractionSteps`). The slice is immutable after declaration; callers receive copies or slices of it. - -### Constants and Defaults - -| Constant | Value | Purpose | -|---|---|---| -| `CodeReducerModelIdEnvKey` | `"CODE_REDUCER_MODEL_ID"` | Overrides `Config.ModelID` when set. | -| `OllamaBaseUrlEnvKey` | `"OLLAMA_BASE_URL"` | Overrides `Config.OllamaBaseURL` when set. | -| `OllamaNumCtxEnvKey` | `"OLLAMA_NUM_CTX"` | Overrides `Config.OllamaNumCtx`; value must parse as a positive integer. | -| `OllamaDefaultBaseURL` | `"http://localhost:11434"` | Fallback base URL when no prior layer supplies one. | -| `OllamaDefaultModelID` | `"ornith:9b"` | Default model identifier. | -| `OllamaDefaultNumCtx` | `8192` | Default context size. | -| `ConfigFileName` | `.code-reducer.yaml` | Filename resolved relative to the current working directory. | - -Three slices are declared at package init and never mutated: - -```go -var DefaultIgnores []string // system-supplied ignore paths -var DefaultIgnoredExtensions []string // system-supplied ignored extensions -var DefaultExtractionSteps []ExtractionStep // ordered extraction phases -``` - -### Public API Surface - -#### Path Resolution and Existence Check - -##### `getConfigPath(cwd string) (string, error)` - -Constructs the full path to `.code-reducer.yaml` by joining `cwd` with `ConfigFileName`. Returns a non-nil error only on I/O failure during construction; normal cases return `nil`. - -##### `ConfigExists(cwd string) bool` - -Returns whether the configuration file exists at the resolved path. Internally calls `os.Stat`; all errors (permission denied, I/O failures, etc.) are swallowed and treated as "does not exist". The function returns no error. - -#### Configuration Loading - -##### `LoadConfig(cwd string) (*Config, error)` - -Reads and parses the YAML configuration file at the resolved path. Error handling: -- File-read failure (`os.ReadFile`) β†’ returned to caller unchanged (pass-through). -- YAML parse failure β†’ wrapped as `"failed to parse yaml config: %w"`. - -Returns a non-nil `*Config` on success; the struct is initialized with zero values before parsing, so callers receive a populated object even when the file contains only valid empty mappings. - -##### `ResolveConfig(repoRoot, modelIdFlag, numCtxFlag string) (*Config, error)` - -Top-level entry point for full configuration resolution: -1. Calls `LoadConfig` to read the YAML file. If the error is a non-existent-file sentinel (`os.IsNotExist(err)`), it swallows and initializes an empty `Config{}`; all other errors propagate wrapped as `"failed to load configuration file:"`. -2. Layers defaults (hard-coded constants) over the loaded config. -3. Layers environment variable overrides using `os.Getenv` for `CODE_REDUCER_MODEL_ID`, `OLLAMA_BASE_URL`, and `OLLAMA_NUM_CTX`. Non-empty values are used; empty strings fall through. Env read errors are swallowed. -4. Applies CLI flags (`modelIdFlag`, `numCtxFlag`). The flag value is parsed via `strconv.Atoi`; only `err == nil && n > 0` is accepted β€” invalid or non-positive values cause the prior layer's value to be retained silently. - -Returns a fully resolved `*Config`. Errors are either pass-through (from `LoadConfig`) or wrapped with `"failed to load configuration file:"`. Notable: if YAML is malformed, the parse error wraps and propagates; otherwise resolution always succeeds for missing files. - -#### Configuration Persistence - -##### `SaveConfig(cwd string, cfg *Config) error` - -Serializes the config struct back to disk using `os.WriteFile` with `0600` permissions. Error handling: -- Write success β†’ returns `nil`. -- Parse failure (marshal YAML) β†’ wrapped as `"failed to marshal yaml: %w"`. -- File-write failure β†’ wrapped as `"failed to write config file: %w"`. - -Writes directly (no temp-file + rename); a partial write can leave a truncated file on disk. Permission normalization is implicit β€” existing files with different permissions will fail the write. - ---- - -## Package: internal/engine β€” Code Documentation Pipeline Engine - -### Responsibility - -Implements a Map-Reduce pipeline that generates and maintains technical documentation for a Go codebase using Ollama-compatible LLM inference endpoints. The engine produces three artifacts per run: `architecture.md` (global overview), `quickstart.md` (onboarding guide), and per-module API summaries under `modules/`. Execution is gated by a repository-level lock to prevent concurrent runs against the same root, and state persistence across invocations is handled by an on-disk metadata cache. - -### Pipeline Orchestration - -#### Runner Entry Point (`runner.go` β€” `Runner.Run`) - -The public entry point for all pipeline invocations. Responsibilities: repository isolation, concurrency guard acquisition, LLM client construction from config, and mode-gated dispatch to the appropriate pipeline method. - -``` -Run(ctx, repoRoot, mode string, onEvent func(Event)) error - β”œβ”€ ensure .gitignore lockfile entry (best-effort; failure logged as warning) - β”œβ”€ acquire repository-level lock via security.AcquireLock(repoRoot) - β”‚ └─ defer lock.Unlock() β€” released on any return path including errors - β”œβ”€ build *LLMClient from r.cfg.ModelID, BaseURL, NumCtx - β”œβ”€ switch mode: - β”‚ case "init": client.RunInit(ctx, repoRoot, r.cfg, onEvent) - β”‚ case "update": client.RunUpdate(ctx, repoRoot, r.cfg, onEvent) - β”‚ default: return fmt.Errorf("unsupported mode: %s", mode) -``` - -The `cfg *config.Config` field is read-only within this file; all mutations to the runner's configuration originate before construction. The lock returned by `security.AcquireLock(repoRoot)` is a local variable with deferred cleanup β€” its internal type and mutual exclusion semantics are not observable from this source alone. - -#### Orchestrator Pipelines (`orchestrator.go` β€” `RunInit`, `RunUpdate`) - -##### Full Run (`RunInit`) - -Executes an all-or-nothing regeneration regardless of cache state: -1. **Setup** β€” Load `.gitignore` via `tools.LoadGitignore(repoRoot)`; merge with user ignore list; load metadata cache from disk via `cache.loadMetadataCache()`. Failures logged as warnings, not returned. -2. **Map Phase** β€” Discover code files via `tools.DiscoverCodeFiles(...)`, compute SHA-256 per file via `computeSHA256(repoRoot, f)`, build directory tree via `buildTree(codeFiles)`. Hash computation errors silently skipped in loops; only stored if hash computed successfully. -3. **Mark All Affected** β€” Recursively flag every node as affected (init starts fresh). -4. **Reduce Phase** β€” Call `synthesizeNode` bottom-up on the tree to produce a root summary string and per-module summaries via LLM calls through `c.CallLLM`. -5. **Generate Standard Docs** β€” Produce `architecture.md` and `quickstart.md` from the root summary via two additional `CallLLM` invocations: one for architecture, one for quickstart. Writes via `tools.WriteFileSafely`; errors wrapped with context (e.g., `"failed to write quickstart.md: %w"`). -6. **Update AGENTS.md** β€” Read existing file via `tools.ReadFileSafely(repoRoot, "AGENTS.md")`. If missing, create with guidelines header. If present but does not contain "AI Agent Guidelines", append separator + new content block. Only writes when content actually differs. Errors: if read fails but write succeeds β†’ `"failed to write AGENTS.md: %w"`; if both fail β†’ wrapped errors chain. -7. **Teardown** β€” Persist metadata cache via `saveMetadataCache(repoRoot, docsDir, cache)`. Failure logged as warning, function always returns nil at end regardless of this failure. - -##### Incremental Run (`RunUpdate`) - -Executes only re-generations for directories whose contents actually changed: -1. **Setup** β€” Load pipeline state from metadata cache (same path as init). -2. **Map Phase** β€” Discover code files, hash each via `computeSHA256`, build tree. -3. **Change Detection** β€” Compare current file hashes against cached hashes in `cache.Files`: - - New file in codebase but absent from cache β†’ `FileChange{Status: "Added"}` - - Existing file with different hash β†’ `FileChange{Status: "Modified"}` - - Cached file no longer in codebase β†’ `FileChange{Status: "Deleted"}` (also removed from cache) -4. **Prune Stale Modules** β€” Remove any module summaries whose directory is no longer active, delete their markdown files on disk via `os.Remove(absModuleFile)` (error assigned to `_`, never logged). -5. **Propagate Affected** β€” Run tree-aware propagation (`propagateAffected`/`determineAffected`) to mark directories affected by changes. Errors from these calls are discarded without capture. -6. **Early Exit** β€” If zero directories are affected β†’ pipeline ends with "up to date" status, no further synthesis occurs. -7. **Conditional Reduce** β€” If root-level change detected or global docs don't exist on disk (`os.Stat(absArch)`/`os.Stat(absQs)`), run full synthesis; otherwise skip and reuse existing `architecture.md`/`quickstart.md`. -8. **Teardown** β€” Save updated cache via `saveMetadataCache(repoRoot, docsDir, cache)`. - -### Directory Tree Construction & Affected Propagation (`tree.go`) - -#### Type Definitions - -| Type | Fields | Purpose | -|------|--------|---------| -| `FileChange` | `Path string`, `Status string` | Models a file operation with path and status (`Added`, `Modified`, `Deleted`). Consumed by `determineAffected`. | -| `DirNode` | `Path string`, `Files []string`, `Children map[string]*DirNode` | Hierarchical tree node holding files at that level and child directories in its subtree. Produced by `buildTree`. | - -#### Functions (all unexported) - -- **`buildTree(codeFiles)`** β€” Converts a flat list of file paths into nested `*DirNode` structure, preserving directory nesting by splitting on `/`. Appends to `DirNode.Files`; populates `DirNode.Children` via map assignment. -- **`propagateAffected(tree, changeset)`** β€” Traverses the tree recursively: if any direct child of the current node is in the changeset, mark it affected. Accumulates affected status upward through ancestors. Returns mutated `affectedDirs`. -- **`determineAffected(tree, changeset)`** β€” Computes module path for each directory via `ToSafeMarkdownFilename`, checks existence on disk via `os.Stat(absModulePath)`, reads metadata cache (`cache.Modules[n.Path] == ""`), marks node affected if any check fails. Only `os.IsNotExist` is acted on; all other errors are silently dropped. - -### LLM Client Abstraction & Response Parsing - -#### Type: `LLMClient` - -```go -type LLMMClient struct { - ModelID string - BaseURL string - NumCtx int - HTTPClient *http.Client // default transport, no custom config (no proxy/redirect/TLS) -} -``` - -Constructed via `NewLLMClient(modelID, baseURL, numCtx)` with a 10-minute HTTP timeout. Fields set once; never reassigned within any method. - -#### Methods on `*LLMClient` - -##### `CallLLM(ctx context.Context, systemPrompt string, messages []Message, jsonFormat bool) (string, error)` - -Full lifecycle: prepends system prompt as first message with role `"system"` β†’ serializes into Ollama `/api/chat` JSON payload with model ID and context window size β†’ non-streaming HTTP POST β†’ response body fully read on 200 OK or truncated at 1 KB via `io.LimitReader(resp.Body, 1024)` on non-OK status. Returns only assistant content string; returns empty string on failure. - -**Error behavior:** - -| Failure path | Error type | Wrapping strategy | -|---|---|---| -| JSON marshal fails in request prep | `fmt.Errorf("failed to marshal request: %w", err)` | Wrapped with `%w` | -| `http.NewRequestWithContext` fails | raw error from `NewRequestWithContext` | **Not wrapped** β€” passed through as-is | -| HTTP Do (network-level failure) | raw error from `c.HTTPClient.Do(req)` | **Not wrapped** | -| Body read fails on 200 OK path | raw error from `io.ReadAll` | **Not wrapped** | -| JSON unmarshal fails on response | `fmt.Errorf("failed to parse response: %w", err)` | Wrapped with `%w` | -| Non-OK status (body truncated at 1 KB) | `fmt.Errorf("ollama api error: status %d, response: %s", ...)` | New error constructed; body discarded if read fails | - -##### `GetBaseSystemPrompt() string` - -Returns the fixed base role definition and defensive guidelines inherited by every LLM call regardless of task type. - -##### `GetDefaultSystemPrompt(command string) string` - -Appends command-specific instructions based on the command string: two named variants exist (`module_synthesis`, `architecture`); any other value receives only the base layer. - -### Response Parsing Layer (`json_parser.go`) - -Internal helper `extractBalancedJSON` is lowercase and excluded from public surface area. - -#### Business Rules - -1. Markdown code fences (` ``` `) surrounding JSON must be stripped before parsing β€” handled by `StripOuterMarkdownFence`. -2. Balanced bracket matching required β€” strings containing `{`, `[`, `}`, `]` literals (including escaped characters like `\n`) must not break extraction logic β€” handled by internal rune-level scanner with escape state tracking inside string literals. -3. When multiple balanced JSON candidates exist in a single response, each must be attempted individually until one succeeds, or the entire trimmed string is tried as a final fallback β€” handled by `CleanJSONResponse`/`UnmarshalJSONResponse`. - -#### Functions - -| Function | Behavior | Error surface | Swallowed? | -|---|---|---|---| -| `StripOuterMarkdownFence(s string) string` | Strips surrounding markdown fences; falls back to trimmed input if none match | None β€” returns string only | N/A (no error return) | -| `CleanJSONResponse(s string) string` | Iterates stripped content, calls `extractBalancedJSON`, returns first successful candidate or original trimmed input unchanged | None β€” returns string only | Yes β€” unbalanced JSON inside response is swallowed silently | -| `UnmarshalJSONResponse(s string, target interface{}) error` | Multi-layered attempt: (1) collect balanced candidates, try each into target; keep last error in `lastErr`; (2) if any succeeded, return nil; otherwise try entire trimmed string; (3) terminal errors β€” `"failed to unmarshal JSON candidates: %w"` or `"no valid JSON found in response"` | Explicit (`error`) | Partial parse failures collected and re-emitted via `%w` on first path; "no valid JSON" covers second path | - -### Synthesis Engine & Chunking (`chunking.go`, `synthesize.go`) - -#### Context-Aware Chunking (all unexported) - -The system processes extracted code/fact items in batches constrained by LLM context limits (`NumCtx * 3`). Three distinct processing modes exist: - -##### Module Synthesis (`reduceInChunks`) - -Synthesizes architectural-level nodes into architecture descriptions using a `"module_synthesis"` system prompt. Each batch represents evidence contributing to understanding the module's design intent. Response passes through `StripOuterMarkdownFence`. Errors wrapped with `"LLM error during synthesis: %w"`. - -##### File Fact Consolidation (`reduceFileFacts`) - -Consolidates extracted facts about a specific step within a file with deduplication and merging. System prompt built locally by concatenating `c.GetBaseSystemPrompt()` + fixed role string. Response also passes through `StripOuterMarkdownFence`. Errors wrapped with `"LLM error during file fact consolidation: %w"`. - -##### Recursive Convergence (`reduceItems`) - -Core business rule: reduce items in batches β†’ collect intermediate results β†’ recursively apply reduction until everything converges into a single output string. Multi-pass approach ensures information is progressively synthesized rather than lost across multiple LLM calls. Termination guaranteed when all data fits within one batch (short-circuits at top with `len(batches) == 1`). Recursion terminates because: -1. Single-batch case short-circuits at top (`len(batches) == 1`) -2. Recursive step processes each batch through itself again with same `maxChars` - -**Graceful Truncation Policy:** When content exceeds the context window, single oversized items get truncated with `\n...[truncated]` marker appended; when multiple items exceed limits collectively, each gets capped at `maxChars / len(items)` characters before being processed as one batch. Infinite-loop guard: if batch size equals item count for all items (can't merge), falls back to per-item truncation using `allowedPerItem = maxChars / len(items)`. Still silent β€” no error returned. - -##### Overlapping Chunk Support (`chunkTextWithOverlap`) - -Utility for splitting raw text into overlapping chunks to maintain context continuity between adjacent segments β€” useful when feeding source material directly into LLMs without pre-processing through the chunking logic. No I/O. Silent swallow: `maxRunes <= 0` β†’ returns full text as single chunk; `overlapRunes >= maxRunes` β†’ clamps to half without warning; final partial chunk reaching end of runes β†’ breaks loop with no error or truncation marker (unlike `reduceItems`). - -#### Hierarchical Module Synthesis (`synthesizeNode`, all unexported) - -Performs recursive synthesis of directory structures into hierarchical summaries β€” transforming raw source code and subdirectory metadata into consolidated documentation at each level of a module tree. - -**Algorithmic Flow:** -1. **Context Check β†’ Cache Reuse Gate**: If current node not in `affectedDirs` AND cached module summary exists, return immediately without processing. Short-circuits unaffected directories. -2. **Recursive Child Synthesis (Bottom-Up)**: Sort children alphabetically, recursively call `synthesizeNode` on each child directory; results collected into map keyed by child name. -3. **File Processing Loop**: For each file in current node: hash-first cache lookup (if precalculated hash exists and matches cached SHA256 β†’ reuse cached facts without reading disk); read + hash fallback otherwise (read via `tools.ReadFileSafely`, compute SHA-256, update cache entry if already existed with that hash); extraction pipeline on cache miss (chunk content using dynamic limit derived from `c.NumCtx Γ— 4 \ No newline at end of file diff --git a/wiki/quickstart.md b/wiki/quickstart.md deleted file mode 100644 index 0a7abc2..0000000 --- a/wiki/quickstart.md +++ /dev/null @@ -1,189 +0,0 @@ -# quickstart.md - -> Last updated: 2026-05-18 -> Maintained by: Code-Reducer agent (auto-generated) - ---- - -## What is Code-Reducer? - -Code-Reducer is a terminal-based documentation-generation tool for Go projects. It produces and maintains technical wikis β€” `architecture.md`, `quickstart.md`, per-module API summaries under `modules/` β€” from your source code by invoking an Ollama-compatible LLM endpoint. - ---- - -## Quick Start - -### Prerequisites - -- **Go 1.21+** (standard library only) -- An **Ollama-compatible** local LLM server running and reachable at the configured base URL -- A working terminal (TTY required for interactive setup; non-TTY requires manual config persistence) - -### First Run β€” Interactive Setup - -```bash -cd -code-reducer setup -``` - -This prompts you for: - -| Prompt | Default | Notes | -|---|---|---| -| LLM Model ID | `ornith:9b` | Overrides via env var `CODE_REDUCER_MODEL_ID` | -| Ollama Base URL | `http://localhost:11434` | Overrides via env var `OLLAMA_BASE_URL` | -| Context Size (num-ctx) | `8192` | Overrides via env var `OLLAMA_NUM_CTX`; must parse as positive int | -| Docs directory | `wiki/` | Custom docs output path | -| Ignore paths/files | none | Comma-separated absolute paths excluded from traversal | -| File extensions to ignore | none | Comma-separated file extensions excluded from traversal | - -Pressing **Enter** on any prompt preserves the existing or default value. Pressing `clear` or `none` wipes list-type inputs entirely. Stdin read failures log to stderr and continue using current values β€” no rollback occurs. - -After setup completes, a `.code-reducer.yaml` file is written with 0600 permissions in your working directory. - -### First Run β€” Manual Configuration (CI / Non-TTY) - -```bash -export CODE_REDUCER_MODEL_ID="ornith:9b" -export OLLAMA_BASE_URL="http://localhost:11434" -export OLLAMA_NUM_CTX=8192 - -code-reducer init -``` - -### Subsequent Runs β€” Incremental Update - -```bash -code-reducer update -``` - -This performs change detection (SHA-256 per file), regenerates only affected modules, and reuses existing `architecture.md`/`quickstart.md` unless the root-level code changed. If no changes are detected, it exits with "up to date" status. - -### Full Regeneration β€” Init Mode - -```bash -code-reducer init -``` - -Regenerates all documentation regardless of cache state: architecture, quickstart, per-module summaries, and `AGENTS.md`. - ---- - -## Configuration Precedence - -The tool resolves configuration through a four-tier pipeline. Each layer overwrites the previous when present; missing layers are skipped transparently. - -| Layer | Source | Mechanism | -|---|---|---| -| 1 (lowest) | Built-in constants | `ornith:9b`, `http://localhost:11434`, `8192` | -| 2 | `.code-reducer.yaml` | Read via `config.LoadConfig`; non-existent file treated as empty config | -| 3 | Environment variables | `CODE_REDUCER_MODEL_ID`, `OLLAMA_BASE_URL`, `OLLAMA_NUM_CTX` (non-empty only) | -| 4 (highest) | CLI flags | `--model-id`, `--num-ctx` (`strconv.Atoi`; non-positive values silently retain prior layer) | - -### Environment Variable Defaults - -```bash -export CODE_REDUCER_MODEL_ID="ornith:9b" -export OLLAMA_BASE_URL="http://localhost:11434" -export OLLAMA_NUM_CTX=8192 -``` - ---- - -## Output Artifacts - -| File | Purpose | Regenerated On | -|---|---|---| -| `architecture.md` | Global architecture overview | Root-level code change, or not present on disk | -| `quickstart.md` | Onboarding guide | Root-level code change, or not present on disk | -| `modules//*.md` | Per-module API summaries | File hash mismatch (Added/Modified) for that module | -| `AGENTS.md` | AI Agent Guidelines header | First run; appended to if existing file lacks the marker | - -Per-file operations (`Added`, `Modified`, `Deleted`) are tracked internally and drive selective regeneration. Stale modules whose directories no longer exist in the codebase are pruned silently β€” their markdown files removed without error logging. - ---- - -## System Architecture Overview - -``` -β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ cmd.RootCmd.Execute() β”‚ -β”‚ (entry point: all lifecycle) β”‚ -β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ init.go / update.go β”‚ -β”‚ (Cobra subcommand registration only) β”‚ -β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ cmd.RootCmd β”‚ -β”‚ β”Œβ”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β” β”‚ -β”‚ β”‚--model-idβ”‚ β”‚--num-ctxβ”‚ β”‚setup/β”‚ β”‚ -β”‚ β””β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”˜ β”‚ -β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ internal/config β”‚ -β”‚ ResolveConfig() β†’ *config.Config β”‚ -β”‚ (4-tier: constants β†’ YAML β†’ env β†’ flags) β”‚ -β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ internal/engine β”‚ -β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ -β”‚ β”‚ runner.go β”‚ β”‚ orchestrator.go β”‚ β”‚ -β”‚ β”‚ Run() │────▢│ RunInit / RunUpdate β”‚ β”‚ -β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ -β”‚ β”‚ -β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ -β”‚ β”‚ tree.go β”‚ β”‚ synthesize.go β”‚ β”‚ chunking.goβ”‚ β”‚ -β”‚ β”‚ buildTree / β”‚ β”‚ reduceInChunksβ”‚ β”‚ reduceFile β”‚ β”‚ -β”‚ β”‚ propagate β”‚ β”‚ reduceItems β”‚ β”‚ facts β”‚ β”‚ -β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ -β”‚ β”‚ -β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ -β”‚ β”‚ LLMClient β”‚ CallLLM β†’ /api/chat β”‚ -β”‚ β”‚ (10m timeout)β”‚ Response parsed via json_parser β”‚ -β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ StripOuterMarkdownFence β”‚ -β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ external packages: β”‚ -β”‚ security.AcquireLock() β†’ repo-level lockfile β”‚ -β”‚ cache.loadMetadataCache() β†’ on-disk state β”‚ -β”‚ tools.{LoadGitignore,DiscoverCodeFiles,...} β”‚ -β”‚ tools.ReadFileSafely / WriteFileSafely β”‚ -β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ -``` - -### How the Modules Interact - -1. **`cmd.RootCmd.Execute()`** resolves configuration and dispatches to `init`, `update`, or `setup`. Cobra output suppression is enabled β€” errors propagate as values; no panics occur within this package. - -2. **`internal/config.ResolveConfig()`** is the single source of truth for runtime configuration. It layers defaults, persisted YAML config (if present), environment variables, and CLI flags in that order. All non-structural state is ephemeral β€” scoped to function calls and returned as values. No package-level mutable state survives beyond initialization. - -3. **`internal/engine.Runner.Run()`** is the pipeline orchestrator. It acquires a repository-level lock (via `security.AcquireLock`) to prevent concurrent runs, constructs an `LLMClient`, then dispatches to either the full regeneration (`RunInit`) or incremental update (`RunUpdate`). The runner's config field is read-only within this file β€” all mutations originate before construction. - -4. **`internal/engine.Orchestrator.Pipeline()`** executes the Map-Reduce logic: - - **Map Phase**: Discovers code files, computes SHA-256 per file, builds a directory tree via `buildTree`. Hash computation errors are silently skipped; only successfully computed hashes are stored. - - **Change Detection** (update mode): Compares current hashes against the metadata cache to classify files as Added/Modified/Deleted. Stale modules whose directories no longer exist are pruned silently (`os.Remove` error ignored). - - **Reduce Phase**: Calls `synthesizeNode` bottom-up on the directory tree to produce per-module summaries and a root summary string via LLM calls. Errors from individual synthesis batches are wrapped β€” single oversized items get truncated with `\n...[truncated]`. Infinite-loop guard falls back to per-item truncation when batch merging stalls. - - **Standard Docs Generation**: Produces `architecture.md` and `quickstart.md` from the root summary via two additional LLM calls. Writes use `tools.WriteFileSafely`; errors are wrapped with context (e.g., `"failed to write quickstart.md: %w"`). - -5. **`internal/engine.tree.go`** builds a nested directory tree from flat file paths, splitting on `/`. Affected propagation traverses the tree recursively β€” if any direct child is in the changeset, ancestors accumulate affected status upward. Module existence checks use `os.Stat`; only `os.IsNotExist` is acted on; all other errors are silently dropped. - -6. **`internal/engine.synthesize.go`** implements context-aware chunking with three modes: - - **Module Synthesis** (`reduceInChunks`): Processes architectural nodes in batches constrained by `NumCtx * 3`. Errors wrapped as `"LLM error during synthesis: %w"`. - - **File Fact Consolidation** (`reduceFileFacts`): Deduplicates and merges extracted facts per step within a file. System prompt built from base + fixed role string. - - **Recursive Convergence** (`reduceItems`): Reduces items in batches β†’ collects intermediate results β†’ recurses until everything converges to a single output. Termination guaranteed: short-circuits at top when `len(batches) == 1`. - -7. **`internal/engine.chunking.go`** provides overlapping chunk support via `chunkTextWithOverlap`, splitting raw text into segments with overlap for context continuity between adjacent LLM calls. Silent swallow rules apply: `maxRunes <= 0` returns full text as single chunk; final partial chunk at end of runes breaks the loop without error or truncation marker. - -8. **LLM Client (`internal/engine.LLMMClient`)** constructs an HTTP client with a default transport (no custom proxy/redirect/TLS config) and a 10-minute timeout. `CallLLM()` prepends the system prompt as role `"system"`, serializes to Ollama `/api/chat` JSON payload, performs non-streaming POST, reads the body fully on 200 OK or truncates at ~1 KB on non-OK status. Response parsing handles markdown code fences, balanced bracket matching with escape state tracking, and multiple JSON candidates in a single response. - -9. **Response Parsing** (`internal/engine.json_parser.go`): `StripOuterMarkdownFence` strips surrounding ``` fences silently. `CleanJSONResponse` iterates stripped content calling an internal balanced-bracket extractor; unbalanced JSON is swallowed β€” the original trimmed input is returned unchanged. `UnmarshalJSONResponse` attempts multiple candidates, keeps the last error, and emits `"no valid JSON found in response"` only after exhausting all options. - -10. **Graceful Shutdown**: SIGINT/SIGTERM handlers are installed via `signal.NotifyContext`. A deferred `stop()` cancels the context; any in-flight work respecting the cancelled context exits cleanly. No error-return path exists for signal handling within this file. - ---- - -## Defensible Design Rules - -- All failure paths return errors propagated to cobra β€” no panics observed. -- Stdio read failures during setup log to stderr and continue with current values; no rollback or restart occurs. -- Lock cleanup always runs via `defer` on any return path, including errors. -- Write operations use safe variants (`ReadFileSafely`, `WriteFileSafely`) where available. -- Permission normalization is implicit β€” existing files with different permissions will fail the write. -- No retry logic: if final persistence fails (e.g., `SaveConfig`), the flow aborts. -- Concurrency primitives are absent in the config package; it is single-threaded by design. \ No newline at end of file From 28ad680f550bebf02be7afcb0071edbf512687a0 Mon Sep 17 00:00:00 2001 From: Juan Ezquerro LLanes Date: Mon, 13 Jul 2026 15:07:30 +0200 Subject: [PATCH 2/5] docs: remove CI workflow and add internal configuration module architecture documentation --- .github/workflows/ci.yml | 26 ----- wiki/modules/internal_config.md | 190 ++++++++++++++++++++++++++++++++ 2 files changed, 190 insertions(+), 26 deletions(-) delete mode 100644 .github/workflows/ci.yml create mode 100644 wiki/modules/internal_config.md diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml deleted file mode 100644 index 5e0fb81..0000000 --- a/.github/workflows/ci.yml +++ /dev/null @@ -1,26 +0,0 @@ -name: CI Pipeline - -on: - push: - branches: [ main, master ] - pull_request: - branches: [ main, master ] - -jobs: - build-and-test: - name: Build & Test - runs-on: ubuntu-latest - steps: - - name: Checkout Repository - uses: actions/checkout@v4 - - - name: Set up Go - uses: actions/setup-go@v5 - with: - go-version: '1.26' - - - name: Run Vet - run: go vet ./... - - - name: Run Tests with Race Detector - run: go test -race -v ./... diff --git a/wiki/modules/internal_config.md b/wiki/modules/internal_config.md new file mode 100644 index 0000000..31c0ba3 --- /dev/null +++ b/wiki/modules/internal_config.md @@ -0,0 +1,190 @@ +# `internal/config` Module Architecture + +## Responsibility + +This module defines the static schema, behavioral contracts, and runtime parameter resolution for an LLM-driven code documentation system called "Code-Reducer". It is not execution logicβ€”it provides configuration inputs that other modules consume to drive a four-phase file analysis pipeline and higher-level document synthesis. Three files constitute the public surface: + +| File | Role | +|---|---| +| `config.go` | Declares constants, struct types (`Config`, `ExtractionStep`), and package-level defaults for extraction prompts. | +| `io.go` | Implements file I/O for YAML persistence with atomic write semantics and permission enforcement. | +| `resolve.go` | Orchestrates configuration resolution across multiple sources using a priority chain. | + +No runtime orchestration lives here. The module is the contract layer consumed by downstream pipeline runners and LLM clients. + +--- + +## Constants (`config.go`) + +All constants are immutable literals; none are reassigned within this file. No mutable state exists in any function scope across the entire `internal/config` package. + +### Environment Variable Keys + +| Constant | Type | Value | +|---|---|---| +| `CodeReducerModelIDEnvKey` | `string` | `"CODE_REDUCER_MODEL_ID"` | +| `OllamaBaseURLEnvKey` | `string` | `"OLLAMA_BASE_URL"` | +| `OllamaNumCtxEnvKey` | `string` | `"OLLAMA_NUM_CTX"` | + +### Default Values + +| Constant | Type | Value | Notes | +|---|---|---|---| +| `OllamaDefaultBaseURL` | `string` | `"http://localhost:11434"` | Local Ollama API default. | +| `OllamaDefaultModelID` | `string` | `"ornith:9b"` | Default inference model. | +| `OllamaDefaultNumCtx` | `int` | `8192` | Default context window size. | +| `DefaultDocsDir` | `string` | `"wiki"` | Output directory for generated docs. | +| `ConfigFileName` | `string` | `".code-reducer.yaml"` | Project-root config filename convention. | +| `configFilePerm` | `int` | `0600` | File permission constant; unexported. | + +--- + +## Structs (`config.go`) + +### `ExtractionStep` + +```go +type ExtractionStep struct { + Name string // yaml:"name" + Prompt string // yaml:"prompt" +} +``` + +- **Name** β€” `string` (yaml tag: `"name"`). Identifies the extraction phase. +- **Prompt** β€” `string` (yaml tag: `"prompt"`). Instruction fed to the LLM for this phase. + +### `Config` + +```go +type Config struct { + ModelID string // yaml:"model_id" + OllamaBaseURL string // yaml:"ollama_base_url" + OllamaNumCtx int // yaml:"ollama_num_ctx" + DocsDir string // yaml:"docs_dir" + SystemPrompt string // yaml:"system_prompt" + ModuleSynthesisPrompt string // yaml:"module_synthesis_prompt" + ArchitecturePrompt string // yaml:"architecture_prompt" + FileFactConsolidationPrompt string // yaml:"file_fact_consolidation_prompt" + ExtractionSteps []ExtractionStep // yaml:"extraction_steps" + Ignore []string // yaml:"ignore" +} +``` + +YAML-mapped fields include model identity, API endpoint, context size, document output directory, system persona prompt, synthesis prompts (module, architecture, file consolidation), extraction steps slice, and ignore list. No field is modified within `config.go`. + +--- + +## Default Extraction Steps (`config.go`) + +### `DefaultExtractionSteps` + +- **Type**: `[]ExtractionStep` +- **Value**: Pre-populated package-level variable with four entries: + - `{Name: "API_SIGNATURES", Prompt: "Task: Extract the public surface area of the file. Output: A strict Markdown list of all exported structs, interfaces, and methods. For each, note the actual input and output types. Ignore internal logic."}` + - `{Name: "BUSINESS_LOGIC", Prompt: "Task: Analyze the business logic and domain concepts. Output: Explain what business rules or domain concepts this file solves. Describe the high-level algorithmic flow. Ignore implementation details."}` + - `{Name: "STATE_AND_CONCURRENCY", Prompt: "Task: Analyze state mutation and concurrency. Output: List all mutable state (global variables, changing struct fields) and what concurrency mechanisms (e.g., sync.Mutex, channels) protect them. If none, state 'No mutable state'."}` + - `{Name: "ERRORS_AND_SIDE_EFFECTS", Prompt: "Task: Analyze side effects and error handling. Output: Detail how this code communicates with the outside world (I/O like network, disk, DB) and how it handles/returns errors (wrap, sentinel, panic)."}` + +This slice is initialized once at package scope and not reassigned anywhere in `config.go`. It serves as the fallback when no YAML config specifies extraction steps. + +--- + +## I/O Operations (`io.go`) + +### Exported Functions + +| Function | Signature | Notes | +|---|---|---| +| **`LoadConfig`** | `func LoadConfig(cwd string) (*Config, error)` | Reads `.code-reducer.yaml` from `cwd`; parses YAML; returns struct or error. | +| **`SaveConfig`** | `func SaveConfig(cwd string, cfg *Config) error` | Writes config via atomic rename pattern with permission enforcement. | +| **`ConfigExists`** | `func ConfigExists(cwd string) bool` | Checks filesystem state for config file presence. | + +### Internal Functions (Not Part of Public Surface) + +- `getConfigPath(string)` β€” lowercase initial, unexported; computes path via `filepath.Join`. No side effects. + +--- + +## Configuration Resolution (`resolve.go`) + +### Exported Function: `ResolveConfig` + +```go +func ResolveConfig(repoRoot, modelIDFlag, numCtxFlag string) (*Config, error) +``` + +Standalone exported function that returns a fully-resolved `*Config`. Does not depend on `config.go` types being defined in this fileβ€”`Config` is referenced from elsewhere. + +### Algorithmic Flow + +#### 1. Load Raw Configuration + +- Calls `LoadConfig(repoRoot)` to read YAML config file rooted at `repoRoot`. +- If the returned error satisfies `os.IsNotExist`, treat it as an empty configuration (zero-value struct). Other errors are propagated up with wrapping: `fmt.Errorf("failed to load configuration file: %w", err)`. + +#### 2. Resolve Extraction Steps + +- Uses configured steps if present in loaded config; otherwise falls back to package-level `DefaultExtractionSteps`. + +#### 3. Resolve Configuration Values by Priority Chain + +Each value is resolved independently using a source-priority chain where later sources override earlier ones: + +| Value | Priority Order (highest wins) | +|---|---| +| Model ID | Default > YAML > Environment Variable (`CodeReducerModelIDEnvKey`) > CLI Flag (`modelIDFlag`) | +| Ollama Base URL | Default > YAML > Environment Variable (`OllamaBaseURLEnvKey`) *(no flag support)* | +| Ollama Context Size | Default > YAML > Environment Variable (`OllamaNumCtxEnvKey`) > CLI Flag (`numCtxFlag`) *(validated: must be positive integer; invalid input yields 0, which fails `n > 0` check and falls through to next source)* | +| DocsDir | Default > YAML | +| SystemPrompt, ModuleSynthesisPrompt, ArchitecturePrompt, FileFactConsolidationPrompt | Default > YAML *(no env or flag override)* | + +#### 4. Return Resolved Config + +- Returns the fully-resolved `*Config` and any error encountered during loading. Final return is `return resolved, nil`. No errors escape except the wrapped config-load failure described above. + +--- + +## Error Handling Strategy (`io.go`) + +### Explicit Errors Returned + +| Function | Failure Path | Behavior | +|---|---|---| +| `getConfigPath` | Returns empty string (no error return path) | Caller must handle empty result. Pure string computation via `filepath.Join`. | +| `ConfigExists` | Returns `bool`; **swallows** the underlying `os.Stat` errorβ€”only reports success/failure, never returns the original error. | Stat error is dropped entirely. No way to distinguish "file not found" from other stat failures. | +| `LoadConfig` | Two distinct failure paths:
1. Raw `os.ReadFile` error β†’ returned as-is (unwrapped).
2. YAML parse error β†’ wrapped with `fmt.Errorf("failed to parse yaml config: %w", err)`. | ReadFile errors are **not** wrappedβ€”caller sees the raw OS-level error from `io.ReadAll`/`OpenFile`. Parse errors get context added via `%w` wrapping (preserves root cause). | +| `SaveConfig` | Returns single `error`. Every failure point is wrapped with a descriptive prefix and `%w`:
- temp file creation
- write to temp file
- sync temp file
- close temp file
- chmod temp file
- rename temp β†’ final config | All errors are **wrapped**, never returned raw. Root cause chain preserved via `%w`. No error swallowing inside this function. | + +### Defer Cleanup in `SaveConfig` + +`SaveConfig` defers `tmpFile.Close()` + `os.Remove(tmpName)`. If the function returns early due to an error, the temp file is still closed and removed from disk (best-effort cleanup). + +--- + +## Side Effects Analysis (`io.go`) + +### External Communication (I/O) + +| Function | Disk/File I/O | Network/DB/etc | Notes | +|---|---|---|---| +| `getConfigPath` | None | None | Pure string computation via `filepath.Join`. No side effects. | +| `ConfigExists` | Reads disk (calls `os.Stat`) | None | Only reads filesystem state to check existence of `.code-reducer.yaml`. | +| `LoadConfig` | Reads disk (calls `os.ReadFile`) then parses YAML from memory | None | Two-step: raw read, then in-memory parse. No external calls between steps. | +| `SaveConfig` | Writes to disk via atomic rename pattern (`tmpFile.Write` β†’ `Sync` β†’ `Close` β†’ `Rename`) | None | Creates temp file in same directory as target config. Persists after all writes + sync complete. | + +**Atomicity note:** `SaveConfig` uses a write-to-temp-then-rename approach to avoid leaving partial files on disk. The rename is the final visible side effect that updates `.code-reducer.yaml`. + +--- + +## Mutable State Analysis (All Files) + +| Variable/Field | Type | Mutated? | Notes | +|---|---|---|---| +| `configFilePerm` | `int` | No β€” constant, immutable by definition. Held as string/integer literal and never reassigned. | +| `Config` struct fields | struct field declarations only | No β€” no instance is created or modified within `config.go`. | +| `DefaultExtractionSteps` | `[]ExtractionStep` (package-level) | No β€” initialized once at package scope with literal slice values and not reassigned anywhere in the file. | +| `resolved` (local `*Config` in resolve.go) | pointer to `Config` struct | Yes β€” fields reassigned inside the function body in priority order: defaults β†’ YAML config (`cfg.*`) β†’ environment variables (`os.Getenv`) β†’ CLI flags. All writes are sequential within a single goroutine; no concurrent access visible here. | +| `resolved.Ignore` | `[]T` (slice) | Yes β€” populated via `mergeAndDeduplicate(cfg.Ignore, nil)`. Local slice, not shared with other goroutines in this file. | +| `resolved.ExtractionSteps` | `[]ExtractionStep` (inferred from usage) | Yes β€” set to `cfg.ExtractionSteps` or `DefaultExtractionSteps`. Local variable only; no concurrent access visible here. | + +**Concurrency Mechanisms:** None detected (`sync.Mutex`, channels, `atomic.*`, goroutines, etc.). All state mutations occur within a single function call without any visible concurrency primitives across the entire package. \ No newline at end of file From f75e860f624115d22e3b108086269a31beddbe96 Mon Sep 17 00:00:00 2001 From: Juan Ezquerro LLanes Date: Mon, 13 Jul 2026 15:40:08 +0200 Subject: [PATCH 3/5] docs: update cmd module documentation and split internal engine details into separate file --- README.md | 117 ++++++++----- wiki/modules/cmd.md | 188 +++++++++------------ wiki/modules/internal_config.md | 218 +++++++++--------------- wiki/modules/internal_engine.md | 288 ++++++++++++++++++++++++++++++++ 4 files changed, 517 insertions(+), 294 deletions(-) create mode 100644 wiki/modules/internal_engine.md diff --git a/README.md b/README.md index 69eb8f7..d4644f4 100644 --- a/README.md +++ b/README.md @@ -10,8 +10,10 @@ Designed specifically for **local development and private LLMs**, Code-Reducer u * **Hierarchical Map-Reduce Pipeline**: Breaks codebase synthesis into a structured Map-Reduce pipeline to document large directories recursively, staying strictly within local LLM context limits. * **Optimized for Private & Local LLMs**: Built specifically to leverage Ollama (e.g., `ornith:9b` or `gemma4:26b`), eliminating expensive cloud API costs and keeping proprietary code local. +* **Fully Customizable Prompting System**: Allows overriding default system prompts, synthesis rules, architecture blueprints, and file fact consolidation directly from YAML configuration. * **Enterprise-Grade Security Sandbox**: Features path traversal guards, atomic process locking, and TOCTOU symlink hijacking defenses for safe workspace operations. * **Fast Incremental Updates**: Uses a filesystem SHA256 hash cache to only re-document modified files, propagating changes upward to minimize LLM calls. +* **Extraction Steps Cache Invalidation**: Automatically detects changes in your extraction steps pipeline and invalidates the cache to ensure documentation accuracy. --- @@ -29,8 +31,10 @@ Compile the executable binary inside the repository root: go build -o code-reducer main.go ``` -### 2. Configure the Tool -Run the interactive wizard to generate `.code-reducer.yaml`: +### 2. Run the Tool (Implicit Setup) +If you run `code-reducer init` or `code-reducer update` in a terminal (TTY) and the `.code-reducer.yaml` configuration file does not exist, the interactive configuration wizard will launch automatically. + +Alternatively, you can manually run the setup wizard first: ```bash ./code-reducer setup ``` @@ -56,20 +60,22 @@ Runs an interactive setup flow in the current directory to generate the `.code-r * LLM Model ID (defaults to `ornith:9b` or reads from existing config) * Ollama Base URL (defaults to `http://localhost:11434`) * Ollama Context Size (defaults to `8192` or reads from existing config) -* Custom files and directories to ignore +* Custom files and directories to ignore (comma-separated) * Documentation output folder name (defaults to `wiki`) ### 2. `code-reducer init` -Scans the repository, builds the hierarchical tree, and generates the initial set of wiki markdown pages. This command creates a metadata cache in `wiki/.metadata.json` containing the baseline metadata file summaries and sets the state commit tracker to `"local"`. -*Note: This command will fail if the project has already been initialized.* +Scans the repository, builds the hierarchical tree, and generates the initial set of wiki markdown pages: +* Generates a metadata cache in `/.metadata.json` containing the baseline metadata file summaries. +* Automatically generates (or appends to) an `AGENTS.md` file in the repository root to guide other AI development agents on how to find and use the generated wiki documentation. +* *Note: This command will fail if the project has already been initialized.* ### 3. `code-reducer update` Detects files modified, added, or deleted since the last documentation run and performs an incremental documentation refresh: * Computes SHA256 hashes of modified files and compares them with the `.metadata.json` cache to extract new technical facts only for files that actually changed. * Rebuilds only the directory-level module summaries (`wiki/modules/.md`) that correspond to changed files. * Skips LLM calls for unchanged directories by reusing the cached summaries in `wiki/.metadata.json`. -* Automatically syncs the global `architecture.md` and `quickstart.md` files if necessary. -*Note: This command requires the project to have been initialized first.* +* Bottom-up propagation: If a file inside a subdirectory changes, the subdirectory is marked "affected" and this state propagates up to the parent directory, rebuilding parents and the root summaries. +* Automatically syncs global files (`architecture.md` and `quickstart.md`) only if the root directory `.` is affected (i.e. top-level structural changes occurred) or if the files are physically missing. --- @@ -86,28 +92,57 @@ model_id: "ornith:9b" ollama_base_url: "http://localhost:11434" # Custom context window size -ollama_num_ctx: 10000 +ollama_num_ctx: 20000 # Directory paths, files, or glob patterns to ignore during scanning ignore: - - ".gitignore" - "README.md" - ".code-reducer.yaml" - - ".code-reducer.lock" - - "code-reducer" + - "go.sum" + - "go.mod" + - "screenshots" + - "examples" # Target directory to write generated markdown documentation docs_dir: "wiki" -# Optional: Customize the LLM extraction pipeline steps -# If omitted, defaults to 4 steps: API_SIGNATURES, BUSINESS_LOGIC, STATE_AND_CONCURRENCY, ERRORS_AND_SIDE_EFFECTS +# System instructions injected to every LLM request +system_prompt: | + You are Code-Reducer, an expert technical writer and code analyzer. Your job is to strictly follow instructions. You do not yap, you do not write filler. + DEFENSIVE RULES: 1. Do NOT use absolute terms ('always', 'never', 'zero') unless explicitly proven. 2. Do NOT guess downstream consequences or invent unhandled paths. If an error is swallowed, just say it is swallowed. 3. Do NOT name standard library packages unless explicitly stated in the source text. 4. Only report facts you are 100% sure about. + +# Synthesis prompt for directory modules +module_synthesis_prompt: |- + Task: Write a technical documentation page for a code module based on the provided list of its internal components. + Rule 1: Group related functions and classes under appropriate Markdown headings. + Rule 2: Explain the responsibility of the module and the data flow. + Rule 3: Keep it highly technical and dense. + +# Synthesis prompt for the global architecture overview +architecture_prompt: |- + Task: Write a global architecture or quickstart document based on the module summaries. + Rule 1: Explain the system boundaries and how the modules interact. + Rule 2: Provide a dense, developer-friendly overview. + +# Prompt used to consolidate chunks of the same file +file_fact_consolidation_prompt: |- + You are a specialized code documentation assistant. + Consolidate, deduplicate and merge the following facts extracted from different chunks of the same file into a single, cohesive summary. + +# Customize the LLM extraction pipeline steps extraction_steps: - - name: "SECURITY_AUDIT" - prompt: "Task: Audit the file for security vulnerabilities.\nOutput: List any potential security risks." + - name: "API_SIGNATURES" + prompt: "Task: Extract the public surface area of the file.\nOutput: A strict Markdown list of all exported structs, interfaces, and methods. For each, note the actual input and output types. Ignore internal logic." + - name: "BUSINESS_LOGIC" + prompt: "Task: Analyze the business logic and domain concepts.\nOutput: Explain what business rules or domain concepts this file solves. Describe the high-level algorithmic flow. Ignore implementation details." + - name: "STATE_AND_CONCURRENCY" + prompt: "Task: Analyze state mutation and concurrency.\nOutput: List all mutable state (global variables, changing struct fields) and what concurrency mechanisms (e.g., sync.Mutex, channels) protect them. If none, state 'No mutable state'." + - name: "ERRORS_AND_SIDE_EFFECTS" + prompt: "Task: Analyze side effects and error handling.\nOutput: Detail how this code communicates with the outside world (I/O like network, disk, DB) and how it handles/returns errors (wrap, sentinel, panic)." ``` ### Precedence Order -Code-Reducer implements a four-tier configuration resolution chain (defined in `internal/config/resolve.go`): +Code-Reducer implements a four-tier configuration resolution chain: ``` [1. CLI Overrides] ──► [2. Environment Variables] ──► [3. YAML Config File] ──► [4. System Defaults] @@ -119,17 +154,15 @@ Code-Reducer implements a four-tier configuration resolution chain (defined in ` * `OLLAMA_BASE_URL` overrides `ollama_base_url` * `OLLAMA_NUM_CTX` overrides `ollama_num_ctx` 3. **YAML File (`.code-reducer.yaml`)**: Read from the repository root. -4. **Defaults**: Hardcoded fallbacks if no other configuration exists: - * Model ID: `ornith:9b` - * Ollama URL: `http://localhost:11434` - * Context Size: `8192` +4. **Defaults**: Hardcoded fallbacks if no other configuration exists. + +### Multi-language & Infrastructure Support +Code-Reducer can be configured to document not only software codebases but also Infrastructure-as-Code (IaC) or cloud topology. You can inspect an example config tailored for Terraform project analysis in [examples/terraform/.code-reducer.yaml](file:///home/arrase/Develop/code-reducer/examples/terraform/.code-reducer.yaml). --- ## πŸ—οΈ Architecture & Technical Deep Dive -This section details the internal mechanics of Code-Reducer for developers wishing to contribute to the codebase or understand its design constraints. - ### 1. Hierarchical Map-Reduce Engine ```mermaid @@ -153,31 +186,26 @@ In `update` mode, the engine dynamically determines which directory nodes are "a * Its cached entry is missing from `.metadata.json` (as part of `MetadataCache.Modules`). * **Propagation**: If a child directory is affected, the status propagates recursively upwards to the parent directory. This triggers a bottom-up rebuild of parent and root summaries. -#### The Map Phase (File Fact Extraction & Overlapping Chunking) +#### Cache Invalidation on Extraction Steps Change +The metadata cache contains a `steps_hash` field representing the SHA256 of the configuration's `extraction_steps` array. During the `update` pipeline, the engine calculates the hash of the current extraction steps. If it differs from `cache.StepsHash`, it clears file facts and module caches, forcing a full regeneration. This ensures that when extraction requirements change, the entire documentation is safely updated. + +#### The Map Phase (Dynamic Chunking with Overlap) For every code file in an affected directory, the engine calculates the `SHA256` of its contents: * **Cache Hit**: Reuses the stored facts string from the cache. -* **Cache Miss**: Analyzes the file using a **Configurable Multi-Prompt Extraction Pipeline**. To handle files that exceed the context window, the engine applies an **Overlapping Chunking Strategy**: - * Large files are split into overlapping fragments (chunks) calculated dynamically based on `c.NumCtx` (typically allocating 75% of context for code and reserving a ~800 character overlap margin to prevent context blindness at chunk boundaries). - * The Map phase loops through each extraction step. For each step, it executes isolated inference on every chunk of the file, injecting rich contextual metadata (like file name, chunk index, and module path) directly into the prompt to guide the LLM. - - By default, it runs 4 steps optimized for smaller ~10B LLMs: - 1. **API_SIGNATURES**: Extracts public signatures and structural contracts. - 2. **BUSINESS_LOGIC**: Infers algorithmic intent and domain rules. - 3. **STATE_AND_CONCURRENCY**: Identifies mutable state, threading models, and sync primitives. - 4. **ERRORS_AND_SIDE_EFFECTS**: Maps I/O boundaries, network calls, and error-handling philosophies. - - These steps can be completely overridden, removed, or expanded in your `.code-reducer.yaml` via the `extraction_steps` array. - -#### The Reduce Phase (Hierarchical Consolidation) -To prevent massive files and folders from blowing out Ollama's context window, Code-Reducer applies a recursive bottom-up consolidation strategy grouped in dynamically sized batches (capped at `c.NumCtx * 3` characters): -* **File-Level Reduce**: If a single file was split into multiple chunks during the Map phase, their extracted facts are recursively deduplicated and consolidated into a unified "Developer Briefing" for the file using a specialized `reduceFileFacts` pipeline. +* **Cache Miss**: Analyzes the file using the configurable `extraction_steps` pipeline. +* **Llm Context-Based File Limits**: Large files are split into overlapping fragments. The engine calculates a dynamic truncation limit: it allocates 75% of `NumCtx` (typically assuming ~4 characters per token) to the file content, reserving the remaining 25% for prompts and output context. An overlap margin (defaults to 800 characters) is used to prevent context blindness at boundaries. +* Isolated inference is run on each chunk for each extraction step. + +#### The Reduce Phase (Hierarchical Consolidation & Truncation Safety) +To prevent massive folders from blowing out Ollama's context window, Code-Reducer applies a recursive bottom-up consolidation strategy grouped in dynamically sized batches (capped at `NumCtx * 3` characters): +* **File-Level Reduce**: If a single file was split into multiple chunks during the Map phase, their extracted facts are consolidated into a unified briefing via `reduceFileFacts`. * **Directory-Level Reduce**: File briefings and child directory summaries are grouped into batches. - * If a directory's components fit into a single batch, they are joined and sent to the LLM with the `module_synthesis` prompt to yield a unified directory summary. - * If they exceed the limit, they are split into sub-batches, reduced independently to intermediate summaries, and recursively merged until a single module summary is achieved. -* Directory summaries are written to `wiki/modules/.md` (root directory resolves to `wiki/modules/root.md`). + * If a directory's components fit into a single batch, they are joined and sent to the LLM with the `module_synthesis_prompt` to yield a unified directory summary. + * If they exceed the limit, they are split into sub-batches, reduced independently, and recursively merged. + * **Truncation Safeguard**: In `reduceItems`, if a single item exceeds the character limit, it is automatically truncated (appending `...[truncated]`) to avoid infinite loops and context buffer overflows. #### Global Synthesis Phase -After reducing the root directory (`.`), the final summary is sent to the LLM to generate two global files: +After reducing the root directory (`.`), the final summary is sent to the LLM to generate: 1. **System Blueprint**: `wiki/architecture.md` (High-level architecture, module boundaries, external integrations). 2. **Developer Quickstart**: `wiki/quickstart.md` (Onboarding guide, configuration guidelines). 3. **AI Agent Guidelines**: Writes guidelines to `AGENTS.md` (or appends to it) to help other incoming agentic developers find and utilize the generated documentation. @@ -190,7 +218,7 @@ The codebase enforces security when accessing local system paths and handling fi #### Path Traversal Guard (`SafeResolve`) Every filesystem operation targeting repository resources passes through `security.SafeResolve`. -1. **Directory Traversal Detection**: It computes the absolute path of the repository root, joins it with the input path, cleans it (using `filepath.Clean`), and obtains the relative path. +1. **Directory Traversal Detection**: It computes the absolute path of the repository root, joins it with the input path, cleans it, and obtains the relative path. 2. **Sanity Check**: If the relative path starts with `..` or there is an error in resolving, it immediately returns a path traversal error, preventing any access to files outside the repository. #### Atomic Process Locking (`security.AcquireLock`) @@ -209,9 +237,8 @@ To serialize execution across multiple terminal windows or background jobs, the Repository scanning is executed using `filepath.WalkDir` coupled with multiple layers of evaluation: 1. **Pruning Subtrees**: Directories that are dot-prefixed (such as `.git` or `.venv`), end in `.egg-info`, or match any ignore rules (from `.gitignore` or configuration) are skipped entirely using `filepath.SkipDir` during traversal, saving CPU cycles. -2. **Ignore Matching Rules**: Ignores loaded from the project's `.gitignore` and specified in the YAML configuration are merged and compiled using a dedicated Gitignore library (`go-gitignore`), ensuring 100% compliance with standard Git semantic rules (like negations and deep globs). +2. **Ignore Matching Rules**: Ignores loaded from the project's `.gitignore` and specified in the YAML configuration are merged and compiled using a dedicated Gitignore library (`go-gitignore`), ensuring 100% compliance with standard Git semantic rules. 3. **Binary Classification (Null-Byte Scanner & Fast-Path)**: - * **Filter Flow**: Files not matching the text extension allowlist (such as unknown suffixes) are checked via binary signature fallback. * **Text Fast-Path**: Common source code extensions (like `.go`, `.js`, `.py`, `.md`) are instantly classified as text, entirely bypassing I/O bottlenecks. * **Fallback Null-Byte Scan**: Unlabelled files are caught by checking the first `1024` bytes for a null byte (`0x00`). If a null byte is found, the file is classified as a binary and skipped. diff --git a/wiki/modules/cmd.md b/wiki/modules/cmd.md index 1fcbb77..eef4e07 100644 --- a/wiki/modules/cmd.md +++ b/wiki/modules/cmd.md @@ -1,144 +1,106 @@ -# Module: `cmd` β€” CLI Command Registration, Lifecycle Management, and Interactive Configuration +# Module: `cmd` β€” CLI Command Orchestration and Entry Point ## Responsibility -The `cmd` package implements the command-layer for the **Code-Reducer** documentation agent. It registers Cobra subcommands (`code-reducer init`, `code-reducer update`, `code-reducer setup`) and delegates execution of each to a shared `executeCommand` handler whose implementation resides outside this package boundary. The package also owns: +The `cmd` module defines the root cobra command (`RootCmd`) for **Code-Reducer**, a documentation agent that writes and maintains project wikis. It registers subcommands (`init`, `update`, `setup`), resolves configuration from merged sources (flags + file), routes engine events to stdout/stderr, handles OS signals for graceful shutdown, and delegates all substantive work to the unshown `executeCommand` entry point. -- Root command construction with environment validation, signal handling, and event-driven output streaming. -- A lifecycle state machine that enforces ordering constraints between subcommands (init before update; setup as prerequisite for run). -- Interactive configuration prompts for first-time or reconfiguration scenarios. +## Data Flow -No exported types, variables, or functions are defined in this package. All identifiers have lowercase first letters and remain internal to `cmd`. +``` +User invokes code-reducer CLI + β”‚ + β”œβ”€β”€ 1. Register subcommands via init() / update.go (package-level) + β”‚ + β”œβ”€β”€ 2. Parse persistent flags (--model-id, --num-ctx) on RootCmd + β”‚ + β”œβ”€β”€ 3. Resolve current working directory; fail if not a git repo + β”‚ + β”œβ”€β”€ 4. Check configuration state + β”‚ β”œβ”€β”€ No config + TTY β†’ RunSetupFlow(repoRoot) + β”‚ └── Config exists β†’ Continue + β”‚ + β”œβ”€β”€ 5. Merge configuration sources: flags + file-based config (config.ResolveConfig) + β”‚ + β”œβ”€β”€ 6. Determine operation mode (init / update / run) and validate state + β”‚ β”œβ”€β”€ init β†’ verify not already initialized + β”‚ β”œβ”€β”€ update β†’ verify project initialized + β”‚ └── other modes β†’ proceed to engine + β”‚ + β”œβ”€β”€ 7. Install signal handler for INT/TERM via signal.NotifyContext + β”‚ + β”œβ”€β”€ 8. Instantiate documentation engine and execute (engine.NewRunner.Run) + β”‚ └── Engine emits typed events during processing + β”‚ β”œβ”€β”€ Status β†’ fmt.Println(ev.Message) [stdout] + β”‚ └── Error β†’ fmt.Fprintf(os.Stderr, "Error: %s\n", ev.Message) [stderr] + β”‚ + └── 9. Return result (success or wrapped error via fmt.Errorf("…: %w", err)) +``` ---- +## Subcommand Registration -## Command Registration Layer (`cmd/init.go`, `cmd/update.go`) +### `init` β€” Documentation Initialization -Both files exist solely as command wiring stubs. Each declares a local `*cobra.Command` variable (unexported) with its `RunE` closure delegating to an external `executeCommand` function: +The `cmd init` subcommand is defined and registered during package initialization in `init.go`. It delegates all processing to the unshown `executeCommand("init")`, which performs repository scanning and generates an initial set of wiki markdown pages. The command itself does not perform any I/O, error wrapping, or logic beyond registration and delegation. -| Identifier | Kind | Behavior | -|---|---|---| -| `initCmd` (`cmd/init.go`) | `*cobra.Command` | Registered under `RootCmd`; `RunE` invokes `executeCommand("init")`. No state mutation after declaration. | -| `updateCmd` (`cmd/update.go`) | `*cobra.Command` | Registered under `RootCmd`; `RunE` invokes `executeCommand("update")`. No state mutation after declaration. | +### `update` β€” Incremental Wiki Update -The domain concept for the init flow is **project initialization** β€” scanning a repository to generate initial wiki documentation (markdown pages). The update flow targets files changed since the last documented commit and updates corresponding wiki pages. Neither file contains business logic; both pass execution to `executeCommand` whose error handling strategy cannot be determined from this source alone because the function is out of scope here. +The `cmd update` subcommand is defined in `update.go`. On invocation, it calls `executeCommand("update")`, which scans the repository for files changed since the last documented commit and regenerates corresponding wiki pages. The actual scanning and regeneration logic resides outside this file; `update.go` only defines the entry point. ---- +### `setup` β€” Interactive Configuration Setup -## Root Command & Lifecycle State Machine (`cmd/root.go`) +The `cmd setup` subcommand is exported as `RunSetupFlow(repoRoot string) error`. It provides an interactive CLI flow that guides users through configuring all application settings, persisting them to `.code-reducer.yaml`. The configuration captures nine fields: model ID, base URL, context size, ignore patterns, docs directory path, and four distinct system prompts (extraction steps, module synthesis, architecture description, file-fact consolidation). -### Core Domain Concepts +The setup flow operates as follows: +1. Attempt to load existing config via `config.LoadConfig(repoRoot)`. Errors are silently swallowed; defaults apply if loading fails. +2. Prompt user sequentially for each setting. Existing values appear in brackets as hints inside the prompt. Empty input or `clear`/`none` retains the existing/fallback value. Ignore patterns accept comma-separated values and replace the entire list on any non-empty input. +3. Assemble final configuration by merging user inputs with preserved existing values (prompts update only if new content is provided; ignores fully override). +4. Persist via `config.SaveConfig(repoRoot, newCfg)`. Success confirms the file path; failure returns a wrapped error `"error saving configuration: %w"`. -The package-level variable `RootCmd` is a `*cobra.Command` initialized with: +## Configuration Resolution and Validation -- `Use` = `"code-reducer"` -- `Short`, `Long` descriptions (contents not exposed here) -- `DisableDefaultHelpCommand` -- `CompletionOptions{DisableDefaultCmd: true}` -- `SilenceUsage`, `SilenceErrors` both set to `true` +### State Initialization -The root command owns four behavioral responsibilities beyond subcommand registration: +Package-level variables are declared and assigned exactly once at initialization: +- `modelIDFlag` (`string`) β€” set via flag parsing on RootCmd. Only read subsequently. +- `numCtxFlag` (`string`) β€” same as above; never reassigned after declaration. +- `RootCmd` (`*cobra.Command`) β€” initialized with a literal; no subsequent writes occur in this file. -1. **Environment validation** β€” verifies the current working directory is a valid Git repository via an external `tools.VerifyGitRepo(repoRoot)` call; failure is hard (no soft fail). -2. **Configuration resolution** β€” merges three sources in order of precedence: persisted config file, CLI flag overrides (`--model-id`, `--num-ctx`), then defaults. The merged result drives all downstream behavior. -3. **Lifecycle state machine** β€” enforces ordering constraints between subcommands (init requires project NOT yet initialized; update requires project IS already initialized). Invalid transitions fail with explicit sentinel messages. -4. **Graceful termination** β€” installs signal handlers for `SIGINT` and `syscall.SIGTERM`. A context is created via `signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)` whose returned `ctx` and `stop` function manage lifecycle; deferring `stop()` cancels the context on return. -5. **Event-driven execution** β€” spawns an engine runner that emits events (status, error, etc.) which are streamed to stdout/stderr in real time rather than collected after completion. The callback passed to `runner.Run` is a side-effect only: it writes events but does not propagate them back as errors. Error state comes from `err != nil` at the end of the run path. +### Configuration Lifecycle Modes -### Mutable State (`cmd/root.go`) +The module enforces two distinct modes for project setup: +- **Init mode**: Creates/initializes documentation infrastructure (wiki structure). Fails if already initialized, requiring `update` to refresh later. +- **Update mode**: Refreshes existing documentation. Fails if the project has not been initialized yet. -| Variable | Type | Mutated? | Notes | -|---|---|---|---| -| `modelIDFlag` (string) | package-level | Once, during flag registration in `init()` via `StringVar` | Not mutated after declaration. | -| `numCtxFlag` (string) | package-level | Once, during flag registration in `init()` via `StringVar` | Same pattern as above. | +### Configuration Resolution Priority -No concurrent access to shared state occurs within this file; no locking mechanism is needed. +The merged configuration is resolved from three sources: +1. Persistent command-line flags (`--model-id`, `--num-ctx`) +2. File-based config (from project directory) +3. These are merged together into a single resolved config object via `config.ResolveConfig(repoRoot, modelIDFlag, numCtxFlag)` ---- +## Signal Handling and Graceful Shutdown -## Interactive Setup Flow (`cmd/setup.go`) +A signal handler captures `INT` (`os.Interrupt`) and `TERM` (`syscall.SIGTERM`) signals via `signal.NotifyContext`. A deferred call to `stop()` ensures the context is cancelled when a signal arrives. The engine runner's returned error is then wrapped as `"documentation run failed: …"`. -### Exported Function +## Error Handling Patterns -| Function | Input(s) | Output | -|---|---|---| -| `RunSetupFlow(repoRoot string)` | `repoRoot string` | `error` | +### Wrap Pattern (100% of observed paths) -No exported structs, interfaces, or methods are defined in this file. All other identifiers (`setupCmd`, `promptStringList`, `promptString`) and type references belong to imported packages or are unexported locals. +Every error originates from a sub-function call, is immediately wrapped with `fmt.Errorf("…: %w", err)`, and propagated to the next level or returned to cobra's main handler. Examples include: +- `os.Getwd()` failure β†’ wrapped as `"failed to get current working directory: …"` +- `tools.VerifyGitRepo` error β†’ passed through unchanged (already a wrapped/typed error) +- `RunSetupFlow` error β†’ passed through unchanged +- `config.ResolveConfig` error β†’ passed through unchanged +- `engine.NewRunner(cfg).Run(...)` result β†’ wrapped as `"documentation run failed: …"` -### Business Rules Resolved by This File +### Sentinel / Early-Return Logic -1. **Initial setup / reconfiguration** β€” supports both first-time configuration (no existing config) and reconfiguration (existing config present). Both paths use the same interactive prompts uniformly. -2. **Configuration persistence with defaults** β€” every configurable field has a built-in default value (`config.OllamaDefault*`, `config.DefaultDocsDir`, etc.). If no prior configuration exists, these defaults become initial values; if a config already exists, its current values serve as hints and are retained unless changed. -3. **LLM model configuration** β€” user must specify an LLM model ID and Ollama base URL for code analysis operations. Both fields accept empty input (falling back to existing/default), allowing partial reconfiguration. -4. **Context size with validation** β€” non-numeric or zero values are rejected, defaulting silently to the existing value rather than failing setup. -5. **Ignore list management** β€” directories/files/patterns to exclude from analysis can be set as comma-separated entries. Special keywords `clear` and `none` reset the list entirely. Existing ignores persist unless explicitly cleared. -6. **Prompt preservation strategy** β€” all four prompt-related fields (System Prompt, Module Synthesis Prompt, Architecture Prompt, File Fact Consolidation Prompt) follow a "preserve existing" rule: only updated if user provides non-empty input; otherwise current values remain unchanged. -7. **Atomic save** β€” after collecting all inputs, the entire configuration is written in one operation (`config.SaveConfig`). No partial writes occur. -8. **Error tolerance during setup** β€” input read errors and invalid conversions are swallowed with warning messages rather than aborting setup. Existing configuration values serve as safety nets throughout. +The function returns immediately if the working directory cannot be obtained (`os.Getwd()` fails). It returns early with a typed error from `tools.VerifyGitRepo` without wrapping. If stdin is not a terminal and config is missing, it returns an explicit sentinel message: `"configuration file … does not exist… Please run 'code-reducer setup'…"`. "Init already done" β†’ returns wrapped error telling user to use `update`. "Not initialized yet" (during update) β†’ returns wrapped error telling user to run `init` first. -### High-Level Algorithmic Flow +### Nil Error Path -1. Load the current working directory; if it fails, propagate the error immediately (wrapped with `%w`). -2. Attempt to load an existing `.code-reducer.yaml` from that directory via `config.LoadConfig(repoRoot)`. Swallowed error β†’ treated as "no prior config". -3. For each configuration field: display a prompt with the current value in brackets as a hint β†’ collect user input via `reader.ReadString('\n')` on a `bufio.Reader(os.Stdin)` created in this function β†’ validate/convert β†’ fall back to existing/default on failure. -4. After all prompts complete, assemble a new `Config` struct combining any accepted inputs with preserved existing values. -5. Write the assembled config atomically via `config.SaveConfig(repoRoot, newCfg)`. On success, confirm completion; on write error, wrap and return as fatal exit for this flow. +If `executeCommand` returns `nil`, the command succeeds silently. No logging or confirmation output is emitted in this file. -### Mutable State (`cmd/setup.go`) +## Concurrency Characteristics -| Variable | Kind | Mutated After Initialization? | Notes | -|---|---|---|---| -| `setupCmd` (package-level) | `*cobra.Command` | No (assigned once in `init()`) | Internal fields (`Use`, `Short`, `Long`, `RunE`) set during construction. Not reassigned after init. | - -No concurrency primitives protect mutable state; no `sync.Mutex`, channels, atomic operations, or other synchronization primitives are present. - ---- - -## Error Handling Strategy - -### Pattern 1: Wrap-and-Return (Fatal) - -Used for unrecoverable setup failures β€” propagated up to Cobra's `RunE`: - -| Failure Point | Mechanism | -|---|---| -| CWD lookup failure in `setup.go` | `fmt.Errorf("failed to get current working directory: %w", err)` | -| Config save failure | `error saving configuration: %w` with `%w` wrapping | -| Init re-run on initialized project (root.go) | Sentinel string error, no wrapping | - -### Pattern 2: Swallow-and-Fallback (Non-fatal) - -Used for user-input and transient I/O issues β€” existing values are preserved: - -- `strconv.Atoi(ctxInputStr)` parse error β†’ falls back to `existingNumCtx` -- `promptString()` stdin read error β†’ prints warning to stderr, returns `existingVal` -- `promptStringList()` stdin read error β†’ prints warning to stderr, returns `existingList` unchanged -- `config.LoadConfig()` error β†’ treated as nil config; all fields default to built-in constants - -### Pattern 3: Sentinel / Hard-Coded Messages (root.go) - -| Condition | Message | -|---|---| -| Init re-run on initialized project | `"the project has already been initialized"` | -| Update without prior initialization | explicit message | -| Missing config for run commands (non-TTY) | soft fail prompting user to run setup first | - -### Pattern 4: Event-Driven Output (root.go runner callback) - -Errors are communicated via engine events (`engine.EventError`) routed to stderr rather than returned as error. Caller reads status through stdout/stderr stream instead of checking return value alone. The callback is side-effect only and does not propagate events back as errors. - -### Pattern 5: Context Cancellation (signal handler + `defer stop()`) - -Graceful shutdown on SIGINT/SIGTERM; runner receives cancelled context which the engine (outside this file) likely translates into an error return. If setup flow fails before reaching signal handler registration, cleanup cost is minimal. No panic observed in any of these files. - ---- - -## Concurrency & Signal Handling - -| Concern | Mechanism | Scope | -|---|---|---| -| Cooperative cancellation | `signal.NotifyContext` with `SIGINT`, `syscall.SIGTERM`; returned `ctx` and `stop()` manage lifecycle; deferring `stop()` cancels on return. | `cmd/root.go` | -| Signal-driven context propagation | Engine runner receives cancelled context which is translated into error by downstream code outside this package. | Cross-package boundary | - -No mutexes (`sync.Mutex`, `sync.RWMutex`) are used in any of these files. No channels or other synchronization primitives appear directly here. Only local/flag variables are mutated once at startup; no concurrent access to shared state occurs within these files, so no locking mechanism is needed. \ No newline at end of file +No mutexes, channels, goroutines, atomics, or other concurrency primitives are used in any of these files. All operations execute sequentially on the main thread. Concurrency relies solely on `context.Context` cancellation via OS signals; no shared-mutable-state synchronization primitives are present. \ No newline at end of file diff --git a/wiki/modules/internal_config.md b/wiki/modules/internal_config.md index 31c0ba3..3062b19 100644 --- a/wiki/modules/internal_config.md +++ b/wiki/modules/internal_config.md @@ -1,190 +1,136 @@ -# `internal/config` Module Architecture +# Module: internal/config β€” Configuration Layer for Code Reducer Synthesis Pipeline ## Responsibility -This module defines the static schema, behavioral contracts, and runtime parameter resolution for an LLM-driven code documentation system called "Code-Reducer". It is not execution logicβ€”it provides configuration inputs that other modules consume to drive a four-phase file analysis pipeline and higher-level document synthesis. Three files constitute the public surface: +This module manages the lifecycle of project-level configuration for a multi-phase LLM analysis pipeline that generates technical documentation from source code. It defines three distinct concerns: (1) compile-time type and constant declarations, (2) file-system operations on `.code-reducer.yaml` with atomic write guarantees, and (3) multi-source value resolution across defaults, YAML, environment variables, and CLI flags. -| File | Role | -|---|---| -| `config.go` | Declares constants, struct types (`Config`, `ExtractionStep`), and package-level defaults for extraction prompts. | -| `io.go` | Implements file I/O for YAML persistence with atomic write semantics and permission enforcement. | -| `resolve.go` | Orchestrates configuration resolution across multiple sources using a priority chain. | +## Data Flow -No runtime orchestration lives here. The module is the contract layer consumed by downstream pipeline runners and LLM clients. +The module's data flow is sequential: `config.go` declares the schema (`Config`, `ExtractionStep`) and compile-time constants; `io.go` provides read/write primitives against a fixed path derived from `cwd`; `resolve.go` composes these into a resolved `*Config` by merging across four priority tiers. No state persists between calls within this moduleβ€”each resolution produces a fresh `*Config` allocation whose fields are populated through the merge chain and returned to callers for consumption elsewhere in the application. --- -## Constants (`config.go`) - -All constants are immutable literals; none are reassigned within this file. No mutable state exists in any function scope across the entire `internal/config` package. - -### Environment Variable Keys +## Configuration Types and Defaults (`config.go`) -| Constant | Type | Value | -|---|---|---| -| `CodeReducerModelIDEnvKey` | `string` | `"CODE_REDUCER_MODEL_ID"` | -| `OllamaBaseURLEnvKey` | `string` | `"OLLAMA_BASE_URL"` | -| `OllamaNumCtxEnvKey` | `string` | `"OLLAMA_NUM_CTX"` | +### Exported Structs -### Default Values - -| Constant | Type | Value | Notes | -|---|---|---|---| -| `OllamaDefaultBaseURL` | `string` | `"http://localhost:11434"` | Local Ollama API default. | -| `OllamaDefaultModelID` | `string` | `"ornith:9b"` | Default inference model. | -| `OllamaDefaultNumCtx` | `int` | `8192` | Default context window size. | -| `DefaultDocsDir` | `string` | `"wiki"` | Output directory for generated docs. | -| `ConfigFileName` | `string` | `".code-reducer.yaml"` | Project-root config filename convention. | -| `configFilePerm` | `int` | `0600` | File permission constant; unexported. | - ---- +**`ExtractionStep`** β€” Carries per-phase identifiers: +- `Name string` β€” Phase identifier (e.g., `"API_SIGNATURES"`, `"BUSINESS_LOGIC"`). +- `Prompt string` β€” The prompt text handed to the LLM for that phase. -## Structs (`config.go`) +**`Config`** β€” Aggregate configuration container referenced by all exported functions: +- `ModelID string` +- `OllamaBaseURL string` +- `OllamaNumCtx int` +- `DocsDir string` +- `SystemPrompt string`, `ModuleSynthesisPrompt string`, `ArchitecturePrompt string`, `FileFactConsolidationPrompt string` β€” First-class prompt templates, all configurable via YAML. +- `ExtractionSteps []ExtractionStep` +- `Ignore []string` -### `ExtractionStep` +### Exported Constants -```go -type ExtractionStep struct { - Name string // yaml:"name" - Prompt string // yaml:"prompt" -} -``` +Environment variable keys (`CodeReducerModelIDEnvKey`, `OllamaBaseURLEnvKey`, `OllamaNumCtxEnvKey`), default values for Ollama instance parameters (`OllamaDefaultBaseURL = "http://localhost:11434"`, `OllamaDefaultModelID = "ornith:9b"`, `OllamaDefaultNumCtx = 8192`), file-system defaults (`DefaultDocsDir = "wiki"`, `ConfigFileName = ".code-reducer.yaml"`), and multiline prompt templates are all package-level constants. -- **Name** β€” `string` (yaml tag: `"name"`). Identifies the extraction phase. -- **Prompt** β€” `string` (yaml tag: `"prompt"`). Instruction fed to the LLM for this phase. +### Exported Variable -### `Config` +**`DefaultExtractionSteps`** β€” Pre-populated slice containing four named phases: `API_SIGNATURES`, `BUSINESS_LOGIC`, `STATE_AND_CONCURRENCY`, `ERRORS_AND_SIDE_EFFECTS`. Initialized at declaration only; no post-initialization mutations occur in this file. -```go -type Config struct { - ModelID string // yaml:"model_id" - OllamaBaseURL string // yaml:"ollama_base_url" - OllamaNumCtx int // yaml:"ollama_num_ctx" - DocsDir string // yaml:"docs_dir" - SystemPrompt string // yaml:"system_prompt" - ModuleSynthesisPrompt string // yaml:"module_synthesis_prompt" - ArchitecturePrompt string // yaml:"architecture_prompt" - FileFactConsolidationPrompt string // yaml:"file_fact_consolidation_prompt" - ExtractionSteps []ExtractionStep // yaml:"extraction_steps" - Ignore []string // yaml:"ignore" -} -``` +### Non-Exported -YAML-mapped fields include model identity, API endpoint, context size, document output directory, system persona prompt, synthesis prompts (module, architecture, file consolidation), extraction steps slice, and ignore list. No field is modified within `config.go`. +Internal helper variable `configFilePerm` (value `0600`) is referenced by `io.go` but not declared here. No exported interfaces or methods exist on these types. --- -## Default Extraction Steps (`config.go`) - -### `DefaultExtractionSteps` - -- **Type**: `[]ExtractionStep` -- **Value**: Pre-populated package-level variable with four entries: - - `{Name: "API_SIGNATURES", Prompt: "Task: Extract the public surface area of the file. Output: A strict Markdown list of all exported structs, interfaces, and methods. For each, note the actual input and output types. Ignore internal logic."}` - - `{Name: "BUSINESS_LOGIC", Prompt: "Task: Analyze the business logic and domain concepts. Output: Explain what business rules or domain concepts this file solves. Describe the high-level algorithmic flow. Ignore implementation details."}` - - `{Name: "STATE_AND_CONCURRENCY", Prompt: "Task: Analyze state mutation and concurrency. Output: List all mutable state (global variables, changing struct fields) and what concurrency mechanisms (e.g., sync.Mutex, channels) protect them. If none, state 'No mutable state'."}` - - `{Name: "ERRORS_AND_SIDE_EFFECTS", Prompt: "Task: Analyze side effects and error handling. Output: Detail how this code communicates with the outside world (I/O like network, disk, DB) and how it handles/returns errors (wrap, sentinel, panic)."}` - -This slice is initialized once at package scope and not reassigned anywhere in `config.go`. It serves as the fallback when no YAML config specifies extraction steps. - ---- - -## I/O Operations (`io.go`) +## Configuration File Lifecycle (`io.go`) ### Exported Functions -| Function | Signature | Notes | -|---|---|---| -| **`LoadConfig`** | `func LoadConfig(cwd string) (*Config, error)` | Reads `.code-reducer.yaml` from `cwd`; parses YAML; returns struct or error. | -| **`SaveConfig`** | `func SaveConfig(cwd string, cfg *Config) error` | Writes config via atomic rename pattern with permission enforcement. | -| **`ConfigExists`** | `func ConfigExists(cwd string) bool` | Checks filesystem state for config file presence. | - -### Internal Functions (Not Part of Public Surface) +| Function | Input | Output | +|----------|-------|--------| +| `ConfigExists(cwd string)` | cwd directory path | `bool` | +| `LoadConfig(cwd string)` | cwd directory path | `*Config`, `error` | +| `SaveConfig(cwd string, cfg *Config)` | cwd, config pointer | `error` | -- `getConfigPath(string)` β€” lowercase initial, unexported; computes path via `filepath.Join`. No side effects. +No exported structs or interfaces are defined in this file. All external type references (`Config`) come from the same package. ---- +### Atomic Write Pattern for SaveConfig -## Configuration Resolution (`resolve.go`) +The save operation does not write directly to the target path. The sequence is: marshal `cfg` into YAML bytes β†’ apply formatting normalization (collapsing single blank lines to double blank lines before known section headers: `system_prompt`, `module_synthesis_prompt`, `architecture_prompt`, `file_fact_consolidation_prompt`, `extraction_steps`, `ignore`) β†’ create a temp file matching `.tmp.*` in the same directory β†’ write normalized YAML into it β†’ sync and close β†’ set permissions via `os.Chmod` β†’ atomically rename temp over original. Readers never observe a partially-written config during save because the old file remains intact until the new one is fully ready to replace it. -### Exported Function: `ResolveConfig` +### Load-Save Roundtrip Integrity -```go -func ResolveConfig(repoRoot, modelIDFlag, numCtxFlag string) (*Config, error) -``` +Loading preserves whatever formatting was originally written; normalization is applied only on write, not read. This decouples the in-memory representation from on-disk formatting expectations. `LoadConfig` returns a fresh `*Config` via unmarshaling with no shared state. Errors from `os.ReadFile` propagate directly; YAML parse failures are wrapped with a `"failed to parse yaml config"` prefix chained via `%w`; marshal failures similarly wrap a descriptive message. -Standalone exported function that returns a fully-resolved `*Config`. Does not depend on `config.go` types being defined in this fileβ€”`Config` is referenced from elsewhere. +### Error Propagation Strategy -### Algorithmic Flow +- **Raw return**: When `os.ReadFile` fails in `LoadConfig`, the OS error is returned unwrapped. +- **Wrap with context (`fmt.Errorf(... %w)`)**: Parse and write failures across all save-path operations (temp create, write, sync, close, chmod, rename). All wrapped calls preserve chain via `%w`. +- **Error-as-boolean sentinel**: `ConfigExists` uses an `err == nil` check instead of returning the error. No wrapping; raw error is reused as a boolean condition. +- **Silent cleanup in defer**: A deferred function calls `tmpFile.Close()` and `os.Remove(tmpName)`. Errors from these cleanup calls are swallowed (not returned). On success, `Close` runs twice on the temp fileβ€”Go allows this safelyβ€”and the defer's `Remove` returns an error that is silently swallowed because the original config now holds the renamed data. -#### 1. Load Raw Configuration +### Disk Operations Summary -- Calls `LoadConfig(repoRoot)` to read YAML config file rooted at `repoRoot`. -- If the returned error satisfies `os.IsNotExist`, treat it as an empty configuration (zero-value struct). Other errors are propagated up with wrapping: `fmt.Errorf("failed to load configuration file: %w", err)`. +| Function | Disk Operations | +|---|---| +| `getConfigPath` (internal) | None; pure computation via `filepath.Join`. | +| `ConfigExists` | One read-only check via `os.Stat`. Error sentinel reused as boolean condition. | +| `LoadConfig` | Two disk reads: `os.ReadFile` then `yaml.Unmarshal`. Missing-file errors propagate raw; parse failures are wrapped. | +| `SaveConfig` | Multiple writes: `os.CreateTemp`, `tmpFile.Write`, `tmpFile.Sync`, `tmpFile.Close`, `os.Chmod`, `os.Rename`. Each step returns its own error; all but the final rename are wrapped. | -#### 2. Resolve Extraction Steps +No network I/O anywhere in this module. No database operations referenced. -- Uses configured steps if present in loaded config; otherwise falls back to package-level `DefaultExtractionSteps`. +--- -#### 3. Resolve Configuration Values by Priority Chain +## Multi-Source Configuration Resolution (`resolve.go`) -Each value is resolved independently using a source-priority chain where later sources override earlier ones: +### Exported Functions -| Value | Priority Order (highest wins) | -|---|---| -| Model ID | Default > YAML > Environment Variable (`CodeReducerModelIDEnvKey`) > CLI Flag (`modelIDFlag`) | -| Ollama Base URL | Default > YAML > Environment Variable (`OllamaBaseURLEnvKey`) *(no flag support)* | -| Ollama Context Size | Default > YAML > Environment Variable (`OllamaNumCtxEnvKey`) > CLI Flag (`numCtxFlag`) *(validated: must be positive integer; invalid input yields 0, which fails `n > 0` check and falls through to next source)* | -| DocsDir | Default > YAML | -| SystemPrompt, ModuleSynthesisPrompt, ArchitecturePrompt, FileFactConsolidationPrompt | Default > YAML *(no env or flag override)* | +| Function | Input Types | Output Types | +|----------|-------------|--------------| +| `ResolveConfig(repoRoot, modelIDFlag, numCtxFlag string)` | `(string, string, string)` | `(*config.Config, error)` | -#### 4. Return Resolved Config +Internal helper `mergeAndDeduplicate` is lowercase and not part of the public surface. Only usage (not definition) of package-level types (`Config`, `OllamaDefaultModelID`, `CodeReducerModelIDEnvKey`, `DefaultExtractionSteps`) is visible here. -- Returns the fully-resolved `*Config` and any error encountered during loading. Final return is `return resolved, nil`. No errors escape except the wrapped config-load failure described above. +### Resolution Priority System ---- +Each configurable field follows its own independent priority chain, lowest to highest: **defaults β†’ YAML config file β†’ environment variables β†’ CLI flags**. The merge-and-deduplicate logic operates on local function scopes with no cross-goroutine visibility and no shared mutable state. -## Error Handling Strategy (`io.go`) +### Field-Specific Priority Chains -### Explicit Errors Returned +| Field | Priority Chain | +|-------|---------------| +| `ModelID` | Default > YAML > Env Var > CLI Flag | +| `OllamaBaseURL` | Default > YAML > Env Var | +| `OllamaNumCtx` (context size) | Default > YAML > Env Var > CLI Flag | +| `DocsDir` | Default > YAML | +| System prompts (`SystemPrompt`, `ModuleSynthesisPrompt`, `ArchitecturePrompt`, `FileFactConsolidationPrompt`) | All start with default, then override with YAML if non-empty | -| Function | Failure Path | Behavior | -|---|---|---| -| `getConfigPath` | Returns empty string (no error return path) | Caller must handle empty result. Pure string computation via `filepath.Join`. | -| `ConfigExists` | Returns `bool`; **swallows** the underlying `os.Stat` errorβ€”only reports success/failure, never returns the original error. | Stat error is dropped entirely. No way to distinguish "file not found" from other stat failures. | -| `LoadConfig` | Two distinct failure paths:
1. Raw `os.ReadFile` error β†’ returned as-is (unwrapped).
2. YAML parse error β†’ wrapped with `fmt.Errorf("failed to parse yaml config: %w", err)`. | ReadFile errors are **not** wrappedβ€”caller sees the raw OS-level error from `io.ReadAll`/`OpenFile`. Parse errors get context added via `%w` wrapping (preserves root cause). | -| `SaveConfig` | Returns single `error`. Every failure point is wrapped with a descriptive prefix and `%w`:
- temp file creation
- write to temp file
- sync temp file
- close temp file
- chmod temp file
- rename temp β†’ final config | All errors are **wrapped**, never returned raw. Root cause chain preserved via `%w`. No error swallowing inside this function. | +### Validation Rules -### Defer Cleanup in `SaveConfig` +- **Numeric fields**: CLI flag and env var values are parsed as integers; a value must be greater than zero to take effect. Invalid or non-positive values fall through silently. +- **String fields**: Empty string is treated as "not set" β€” the lower-priority default remains active. +- **Config file absence** (`config.yml`): Not an error. If missing, an empty `Config{}` struct is created and defaults are applied. Other parse errors are surfaced. -`SaveConfig` defers `tmpFile.Close()` + `os.Remove(tmpName)`. If the function returns early due to an error, the temp file is still closed and removed from disk (best-effort cleanup). +### Algorithmic Flow for ResolveConfig ---- +1. Load YAML config β€” if it exists and parses successfully; otherwise start with an empty struct (missing-file case). +2. Resolve extraction steps β€” use YAML value if present, else fall back to `DefaultExtractionSteps`. +3. Deduplicate ignore list from the loaded config using a map-based seen-set while preserving first-seen order. +4. For each field, apply its specific priority chain: start with built-in default constant β†’ override with YAML (if non-empty) β†’ override with environment variable (if set and valid for numeric types) β†’ override with CLI flag (highest priority; numeric types require successful integer parse and positive value). +5. Return the fully resolved `Config` struct, or an error if the YAML file fails to parse (excluding missing-file case). -## Side Effects Analysis (`io.go`) +### Error Handling in ResolveConfig -### External Communication (I/O) +- **Swallowed path**: When `os.IsNotExist(err)` is true during config loading, defaults apply with no error returned. +- **Propagated path**: Any other error (parse failure, permission denied) is wrapped with a prefix and returned as `(*Config, nil, error)` β€” the config pointer is `nil`. +- **Silent branches**: String comparisons (`!= ""`, `> 0`) short-circuit without errors. `strconv.Atoi` failures are caught inline; env/flag values simply not used. Map lookups for deduplication have no error path. -| Function | Disk/File I/O | Network/DB/etc | Notes | -|---|---|---|---| -| `getConfigPath` | None | None | Pure string computation via `filepath.Join`. No side effects. | -| `ConfigExists` | Reads disk (calls `os.Stat`) | None | Only reads filesystem state to check existence of `.code-reducer.yaml`. | -| `LoadConfig` | Reads disk (calls `os.ReadFile`) then parses YAML from memory | None | Two-step: raw read, then in-memory parse. No external calls between steps. | -| `SaveConfig` | Writes to disk via atomic rename pattern (`tmpFile.Write` β†’ `Sync` β†’ `Close` β†’ `Rename`) | None | Creates temp file in same directory as target config. Persists after all writes + sync complete. | +### Always-On Return -**Atomicity note:** `SaveConfig` uses a write-to-temp-then-rename approach to avoid leaving partial files on disk. The rename is the final visible side effect that updates `.code-reducer.yaml`. +The function always returns a non-nil tuple `(*Config, error)` β€” either a populated config with `nil` error, or a `nil` config with an error string. No panics anywhere in this file. --- -## Mutable State Analysis (All Files) - -| Variable/Field | Type | Mutated? | Notes | -|---|---|---|---| -| `configFilePerm` | `int` | No β€” constant, immutable by definition. Held as string/integer literal and never reassigned. | -| `Config` struct fields | struct field declarations only | No β€” no instance is created or modified within `config.go`. | -| `DefaultExtractionSteps` | `[]ExtractionStep` (package-level) | No β€” initialized once at package scope with literal slice values and not reassigned anywhere in the file. | -| `resolved` (local `*Config` in resolve.go) | pointer to `Config` struct | Yes β€” fields reassigned inside the function body in priority order: defaults β†’ YAML config (`cfg.*`) β†’ environment variables (`os.Getenv`) β†’ CLI flags. All writes are sequential within a single goroutine; no concurrent access visible here. | -| `resolved.Ignore` | `[]T` (slice) | Yes β€” populated via `mergeAndDeduplicate(cfg.Ignore, nil)`. Local slice, not shared with other goroutines in this file. | -| `resolved.ExtractionSteps` | `[]ExtractionStep` (inferred from usage) | Yes β€” set to `cfg.ExtractionSteps` or `DefaultExtractionSteps`. Local variable only; no concurrent access visible here. | +## Concurrency and Mutability Summary -**Concurrency Mechanisms:** None detected (`sync.Mutex`, channels, `atomic.*`, goroutines, etc.). All state mutations occur within a single function call without any visible concurrency primitives across the entire package. \ No newline at end of file +Across all three files: no `sync.Mutex`, channels, atomics, or other synchronization primitives appear. No package-level variables are modified after initialization. All functions operate on caller-provided arguments and return results. The module is designed for single-call composition with no shared mutable state requiring protection. \ No newline at end of file diff --git a/wiki/modules/internal_engine.md b/wiki/modules/internal_engine.md new file mode 100644 index 0000000..890f448 --- /dev/null +++ b/wiki/modules/internal_engine.md @@ -0,0 +1,288 @@ +# internal/engine β€” Documentation Synthesis Engine + +## Module Responsibility & Data Flow + +The `internal/engine` package implements a recursive, context-window-aware LLM-based documentation synthesis engine. It operates in two modesβ€”**init** (full generation from scratch) and **update** (incremental refresh)β€”orchestrating file discovery, change detection, chunked fact extraction, hierarchical synthesis, and persistent cache management for all intermediate state. + +Data flow: `Runner.Run` β†’ acquires lock β†’ instantiates LLM client + orchestrator β†’ dispatches to `RunInit` or `RunUpdate` β†’ discovers code files via `tools.DiscoverCodeFiles` β†’ classifies changes (Added/Modified/Deleted) against persisted `MetadataCache` β†’ builds directory tree β†’ propagates affected flags upward β†’ recursively synthesizes each node via `synthesizeNode` (per-chunk LLM extraction β†’ fact consolidation β†’ reduction) β†’ writes module docs to disk β†’ persists cache. + +All external communication is delegated: network calls go through the injected `llmCaller.CallLLM`; filesystem I/O goes through internal tool wrappers (`tools.ReadFileSafely`, `tools.WriteFileSafely`); path resolution uses `security.SafeResolve`. No mutexes, channels, or atomic operations are used anywhere in this package. + +--- + +## Constants & Types (constants.go) + +### Constants + +| Constant | Value | Purpose | +|---|---|---| +| `defaultHTTPTimeout` | 10 min | Default HTTP client request timeout | +| `maxErrorBodyBytes` | 1024 bytes | Maximum size allowed in error response bodies | +| `defaultChunkOverlap` | 800 chars | Overlap between consecutive chunks during streaming/processing | +| `minNumCtxFloor` | 512 tokens | Minimum context window floor for processing | +| `contextWindowAllocRatio` | 0.75 | Ratio used when allocating context windows (reserves 75% of available space) | +| `maxCharsMultiplier` | 3 | Multiplier applied to character counts (scales token estimates or max output limits) | +| `metadataFileName` | `.metadata.json` | Persistent metadata file name for the engine's state | +| `agentsFileName` | `AGENTS.md` | Agent definitions/configuration file | +| `defaultDirPerm` | 0755 | Default directory permission mode | + +### Type: `LogEventFunc` + +Function type alias with signature `func(EventType, string)`. Callback interface for logging events; no business rules defined here. + +--- + +## LLM Client Communication (client.go) + +### `Message`, `llmClient`, `CallLLM` + +Unexported except `Message` struct (`Role string`, `Content string`). The package-private `llmClient` carries: model ID, base URL, context count, HTTP client pointer. All set once during construction; never modified afterward within this file. No mutexes or channels used. + +### Algorithmic Flow + +1. **Prepare payload**: Assemble request body by prepending `systemPrompt` as a system-role message before any user-supplied messages, serialize as JSON to Ollama `/api/chat`. +2. **Send HTTP request**: Single synchronous POST with fixed timeout; no retries (fail-fast policy). +3. **Handle response**: On 200 OK β†’ deserialize body β†’ extract `message.content` field. Non-2xx status code β†’ error string containing both HTTP status and raw response payload. +4. **Return result**: Extracted text content or wrapped error describing what went wrong. + +### Error Handling Patterns + +| Path | Wraps? | Notes | +|---|---|---| +| Marshal failure | βœ… `%w` | Good; caller can inspect via `errors.Is` | +| HTTP request construction | ❌ raw | Unwrapped; context lost for callers | +| Network/transport error | ❌ raw | Unwrapped; timeout vs DNS vs connect indistinguishable to caller | +| Success-path body read | ❌ raw | Unwrapped (rare) | +| JSON unmarshal failure | βœ… `%w` | Good | +| Non-OK status response | ❌ custom message | Discards read error (`_`); uses undefined `maxErrorBodyBytes` constant | + +No retry logic. `defaultHTTPTimeout` and `maxErrorBodyBytes` referenced but not defined in this file; if they come from another package, callers have no visibility into their values or how they are set. No context propagation check between `Do()` and `Body.Close()`. + +--- + +## Runner & Entry Point (runner.go) + +### `Runner` / `Run` + +`Runner` wraps a config pointer and exposes a single `Run(ctx, repoRoot, mode, onEvent)` method that is the only public entry point into the pipeline. It performs four sequential operations: + +1. **Gitignore lockfile maintenance** β€” attempts to append the lockfile path to `.gitignore`. Failure is logged via `onEvent(EventStatus, ...)` and execution continues; no error returned. +2. **Repository lock acquisition** β€” calls `security.AcquireLock(repoRoot)`. On failure, returns wrapped error: `"failed to acquire repository lock: %w"`. No partial work proceeds without the lock. +3. **Mode dispatch** β€” constructs an LLM client from config fields (`cfg.ModelID`, `cfg.OllamaBaseURL`, `cfg.OllamaNumCtx`) and instantiates an `orchestrator`. Dispatches to either `RunInit` or `RunUpdate`; any other mode returns `"unsupported mode: %s"`. +4. **Deferred lock release** β€” `defer lock.Unlock()` runs after the pipeline completes, regardless of success or failure. + +The `Runner.cfg` field is set once in `NewRunner(cfg)` and never mutated within this file. The pointer type allows external reassignment but no such mutation occurs here. + +--- + +## Orchestrator & Pipeline Modes (orchestrator.go) + +### `EventType`, `Event`, `orchestrator` + +Unexported struct with two exported constants on the `EventType` type alias: `EventStatus = "status"` and `EventError = "error"`. The unexported `orchestrator` struct carries a method receiver; its public surface is empty. + +### RunInit β€” Cold Start Flow + +``` +1. setupPipeline() β†’ discover files, compute hashes, load .gitignore (swallowed), load cache (swallowed) +2. Mark every directory as affected (full regeneration) +3. Synthesize root summary from full tree via LLM calls +4. Generate architecture.md + quickstart.md using root summary +5. Write agent guidelines file (append if marker missing, overwrite otherwise) +6. Persist cache to disk β†’ complete +``` + +### RunUpdate β€” Incremental Refresh Flow + +``` +1. setupPipeline() β†’ discover current files, compute hashes +2. If extraction steps changed β†’ invalidate entire cache (.StepsHash mismatch resets Files and Modules maps) +3. Classify all discovered files into Added/Modified/Deleted buckets by comparing current vs cached SHA-256 hashes +4. Build directory tree from allowed (existing + new) files only +5. Purge stale module summaries: for any `cache.Modules` path not represented in the live tree, delete both the cache entry and the physical `.md` file (`_ = os.Remove(...)` β€” error swallowed). +6. Compute affected directories; propagate upward through parent dirs. +7. If `len(affectedDirs) == 0` β†’ return early (no-op, docs are up to date). +8. Synthesize root summary from the affected region via LLM calls. +9. Re-generate architecture.md + quickstart.md only if `.` is in `affectedDirs`, or either file does not exist on disk, or resolution fails (treated as "not exists"). +10. Persist cache β†’ complete. +``` + +### Pipeline Context (`pipelineContext`) + +Unexported struct carrying: LLM client pointer, repo root string, config pointer, `*MetadataCache` pointer, affected directory set map, precomputed file hashes map, and event logger callback. Constructed fresh per invocation; no shared instances exist across goroutines in this file. + +--- + +## Cache Persistence Layer (cache.go) + +### `MetadataCache`, `FileCacheEntry` + +Two structs with no methods: +- **`FileCacheEntry`** β€” pairs a SHA256 digest string (`sha256`) with extracted facts string (`facts`). Stored in `MetadataCache.Files map[string]FileCacheEntry`. +- **`MetadataCache`** β€” carries version int, steps hash string, files map, and modules map (module dir β†’ synthesized summary). + +### IsInitialized(repoRoot, docsDir) bool + +Returns true only if the cache file exists AND is readable in one call. Reuses the existence check from `loadMetadataCache`. No error return. + +### saveMetadataCache(repoRoot, docsDir, cache) error + +Serializes to `{docsDir}/{metadataFileName}` with indented JSON (`json.MarshalIndent(cache, "", " ")`). Marshaling errors propagate directly; write errors pass through `tools.WriteFileSafely`. The version field is pinned to the current value on every save. + +### Version Compatibility Guard + +On load: if `cache.Version != currentCacheVersion` (currently 1), returns a fresh zero-valued `MetadataCache{}` with nil error. No warning, no panicβ€”caller sees success with an empty cache. This prevents stale metadata from corrupting future runs without explicit migration handling. + +### Fault-Tolerant Initialization + +If the file does not exist (`os.ErrNotExist` sentinel matched via `errors.Is`), returns a populated zero-valued cache with nil error. If it exists but cannot be read or unmarshaled, returns wrapped error `"failed to read metadata cache: %w"`. Callers can distinguish "not yet initialized" from "corrupt data." + +--- + +## Chunking & Reduction Engine (chunking.go) + +### reduceWithLLM, reduceInChunks, reduceFileFacts + +All unexported; package-private. Implements a **context-aware recursive reduction** for LLM-based synthesis: + +1. **Split**: Input list divided into batches whose combined character count fits within `contextSize Γ— maxCharsMultiplier`. +2. **Reduce each batch**: Each batch sent to LLM via domain-specific prompt β†’ one summary string per batch. If only one batch exists, returned directly; otherwise summaries concatenated and fed back into step 1 recursively until a single result remains. +3. **Graceful truncation**: Items exceeding `maxChars` are truncated with `"...\n...[truncated]"`. If every item exceeds the per-item budget (`allowedPerItem`), all items uniformly truncated to avoid infinite loops. +4. **Context cancellation**: Checked at entry points and before each recursive reduction round; cancellation errors propagate as-is. + +### chunkTextWithOverlap (unused in this file) + +Text-splitting utility with overlap between adjacent chunks. Appears available for reuse elsewhere but not referenced by any function here. + +--- + +## Synthesis Core (synthesize.go) + +All types and functions unexported (`pipelineContext`, `synthesizeNode`). Package-private. + +### Recursive Directory Synthesis + +A directory's summary is derived by recursively synthesizing children (subdirectories) and files together. Final result stored in cache under `cache.Modules[node.Path]` and written to disk at `docs/modules/`. + +### Caching Strategy with Hash-Based Invalidation + +- **Module-level**: Stores synthesized summary per directory path. Reused when affected status indicates no changes (short-circuit return). +- **File-level**: Stores SHA256 hash + extracted facts in `cache.Files[f]`. Cache hit determined by comparing precomputed or read file hash against cached entry; only re-reads on miss. + +### Multi-Step Extraction Pipeline Per File Chunk + +1. System prompt augmented with current extraction step's prompt. +2. Single LLM call extracts facts from chunk β†’ one fact per step. +3. All steps produce respective facts for the same file β†’ consolidated via `reduceFileFacts`. + +### Synthesis Order & Assembly + +1. File facts (sorted child name order). +2. Child subdirectory summaries (after all files processed). +3. Combined list reduced to final summary via `reduceInChunks` at directory level. + +### Affected-Dir Pruning + +If a directory path has no affected status AND a cached module summary exists β†’ short-circuit return cached value without reprocessing children or files. + +--- + +## Tree & Change Detection (tree.go) + +### Exported Types: `FileChange`, `DirNode` + +- **`FileChange`** β€” `Path string`, `Status string` ("Added", "Modified", "Deleted"). +- **`DirNode`** β€” `Path string`, `Files []string`, `Children map[string]*DirNode`. + +### Three-Phase Algorithm + +1. **Build tree**: Given raw file paths, split each on `/`, create intermediate `DirNode` entries for subdirectories, attach files to leaf nodes via `buildTree`. +2. **Determine affected directories from changes**: For each input change, mark directly changed file as affected (if exists in current set). When a file is Deleted, mark parent directory as affected. Walk recursively and check additional conditions: if node path matches `docs/modules/` but no such doc exists on disk β†’ affected; if node has empty cached metadata entry β†’ affected. +3. **Propagate status upward**: Recursive walk of `DirNode` tree propagates any `affected` flag from children to parents so a parent is considered affected if any child is. + +--- + +## JSON Parser Utility (json_parser.go) + +### stripOuterMarkdownFence(input) string + +Single-pass parser that strips markdown or JSON code fences from strings: +1. Trims whitespace from input. +2. Attempts regex match against pattern capturing triple backticks with 3+ chars, optionally followed by `markdown` or `json`, then all content between opening and closing fences (group 1). +3. If matched β†’ returns trimmed captured inner content. +4. If no match β†’ returns original trimmed input unchanged. + +No error return parameter. `regexp.MustCompile` panics on invalid pattern at package initβ€”unhandled within this file, propagates to any caller importing the package. + +--- + +## Utility Functions for Filename Generation and Event Logging (utils.go) + +### Responsibility + +This file provides two unexported utility functions supporting the engine's documentation generation pipeline and optional event logging instrumentation. Neither function communicates with external systems, modifies shared state, or returns errors. All operations are pure in-memory transformations. + +--- + +### toSafeMarkdownFilename β€” Module Path to Markdown Filename Conversion + +**Signature:** `func toSafeMarkdownFilename(path string) string` (inferred from usage; exact signature not exported) + +#### Purpose + +Transforms a module path into a safe markdown documentation filename, enforcing the engine's implicit one-to-one mapping contract between modules and their `.md` documentation files. The root module is special-cased to default to `root.md`. + +#### Data Flow + +1. Input: a raw string representing a module path (e.g., `"internal/engine"`). +2. Character substitution: every `/` character in the input is replaced with `_`. +3. Extension appended: `.md` is concatenated to produce the final filename. +4. Output: a single `string` containing the resulting markdown filename. + +#### Behavior Notes + +- No filesystem, network, database, or other external I/O occurs during execution. +- The function returns only a result string; no error type is returned. Invalid input characters (anything other than `/`) are passed through unmodified β€” callers receive no escape hatch for detecting malformed paths. +- Pure: no mutable state, no package-level globals modified, no struct fields touched. + +--- + +### makeLogEvent β€” Optional Event Callback Adapter + +**Signature:** `func makeLogEvent(onEvent LogEventFunc) func(EventType, message string)` (inferred from usage; `EventType`, `LogEventFunc` referenced but not defined in this file) + +#### Purpose + +Wires an optional callback into a logging pipeline. When invoked with an event type and message pair, it either forwards the call to the registered handler or silently discards it. This makes the logging subsystem purely additive β€” no required consumer exists for the engine's log events; instrumentation is opt-in only. + +#### Data Flow + +1. Input: a user-provided `onEvent` callback of type `LogEventFunc`. +2. Closure construction: returns a new function accepting `(EventType, message)` pairs. +3. Invocation behavior: if `onEvent == nil`, the returned closure does nothing when called β€” no panic, no error, no log output. If `onEvent` is non-nil, the call is forwarded to it with the provided type and message arguments. + +#### Behavior Notes + +- No I/O anywhere in this flow. +- Pure: no mutable state, no package-level globals modified. +- Silent failure path: when `onEvent == nil`, events are swallowed without notice or error wrapping. +- The returned function is always valid (non-nil), so callers can invoke it safely regardless of whether a handler was registered. + +--- + +### Referenced Types (Defined Elsewhere) + +The following types are referenced in this file but not defined within it: + +| Type | Role | +|---|---| +| `EventType` | Categorizes log events into discrete types; carried as the first argument to event handlers and consumed by the returned closure from `makeLogEvent`. | +| `LogEventFunc` | Function signature for user-provided event handlers. Expected to accept `(EventType, message string)` or equivalent parameters. | +| `Event` (struct) | Domain model with at least two fields: `Type` (`EventType`) and `Message` (`string`). Events are tracked as discrete, typed records carrying descriptive payloads. | + +--- + +### Concurrency & State Summary + +- **Mutable state:** None. Both functions operate entirely on their inputs and return values without touching package-level variables or struct fields. +- **Concurrency mechanisms:** None. No mutexes, channels, atomic operations, or synchronization primitives are used in this file. \ No newline at end of file From a409b58833d8a4338abb58239518bee42a625956 Mon Sep 17 00:00:00 2001 From: Juan Ezquerro LLanes Date: Mon, 13 Jul 2026 15:42:37 +0200 Subject: [PATCH 4/5] docs: add internal_security.md documentation covering path containment, process locking, and gitignore management --- wiki/modules/internal_security.md | 165 ++++++++++++++++++++++++++++++ 1 file changed, 165 insertions(+) create mode 100644 wiki/modules/internal_security.md diff --git a/wiki/modules/internal_security.md b/wiki/modules/internal_security.md new file mode 100644 index 0000000..ed9bb86 --- /dev/null +++ b/wiki/modules/internal_security.md @@ -0,0 +1,165 @@ +# internal/security β€” Path Containment & Process Mutual Exclusion + +## Module Responsibility + +This module enforces two non-negotiable invariants for any code-reducer session: (1) the working directory must remain strictly within the repository root boundary, and (2) only one process may hold an exclusive lock on the shared resource at any given moment. It also maintains a `.gitignore` entry that excludes the lockfile from version control while ensuring it is automatically tracked if absent. + +The module communicates with the outside world indirectly: `security.go` performs filesystem I/O and returns sentinel error values to callers; those sentinels are defined in `errors.go`. No panic recovery, no error wrapping beyond `%w`, no silent swallowing of non-trivial failures occurs within these files. + +## Sentinel Error Contract (`errors.go`) + +### Package-Level Variables + +| Identifier | Type | Purpose | +|---|---|---| +| `ErrPathTraversal` | `error` | Returned when a resolved path escapes outside the repository root boundary | +| `ErrLockHeld` | `error` | Returned when another code-reducer process holds an exclusive file lock on the shared resource | + +Both variables are initialized once via `errors.New` and remain read-only throughout their scope. No mutable state, no concurrency primitives, no panic handling exists in this file. Callers detect violations by matching against these sentinels using `errors.Is(err, ErrPathTraversal)` or equivalent patterns; because the errors are unwrapped as-is (no wrapping), identity is preserved directly through the error chain. + +### Domain Concepts + +- **Path Traversal Violation** (`ErrPathTraversal`) β€” signals when a resolved path escapes outside the repository root boundary. This is a classic path traversal / directory escape guardrail. +- **Lock Conflict** (`ErrLockHeld`) β€” signals when another process already holds an exclusive file lock, indicating contention on a shared resource. + +## Lock Acquisition and Release (`security.go` β€” `SimpleLock`) + +### Struct Definition + +```go +type SimpleLock struct { + lockPath string + file *os.File + mu sync.Mutex + closed bool +} +``` + +| Field | Type | Semantics | +|---|---|---| +| `lockPath` | `string` | Absolute path to the lockfile on disk | +| `file` | `*os.File` | Open file descriptor for the lockfile, or nil if not acquired | +| `mu` | `sync.Mutex` | Protects internal struct fields during `Unlock()` | +| `closed` | `bool` | Marks whether the lock has been released and the file descriptor closed | + +### `*SimpleLock.Unlock()` + +Closes the lock file descriptor and removes the lockfile in a thread-safe manner. + +**Idempotency contract:** Calling `Unlock()` on an already-closed lock returns `nil`. This is intentional, not accidental. + +**Error handling inside the method:** +- If `l.closed == true` β†’ return `nil` +- Close the file descriptor (`os.File.Close`) β†’ error captured in local variable; if close fails, set `file = nil` so a subsequent close becomes a no-op +- Remove the lockfile via `os.Remove` +- If *both* close and remove fail: remove error overwrites prior close error only when close returned without error. If close failed first, any remove error is ignored. This means the final returned error reflects whichever operation last produced an error, not necessarily the most severe one β€” an intentional design choice worth flagging for callers. +- If `os.IsNotExist(err)` on remove β†’ silently accepted as normal idempotent cleanup + +## Path Resolution (`security.go` β€” `SafeResolve`) + +### Signature + +```go +func SafeResolve(repoRoot, inputPath string) (string, error) +``` + +**Responsibility:** Resolve any input path strictly inside the repository root. The function performs a multi-step containment check to defeat symlink-based escape vectors and physical directory traversal. + +### Algorithmic Steps + +1. Call `filepath.Abs` on `inputPath` +2. Evaluate symlinks on the existing ancestor via `EvalSymlinks` β€” prevents symlink-based escape from parent directories that exist but are symbolic links +3. Walk up from the target until a physically-existing directory is found, then re-evaluate symlinks on that ancestor via `EvalSymlinks` again +4. Rejoin with the remaining suffix components after the physical ancestor boundary +5. Verify final result is under `resolvedRoot` via relative path check (`filepath.Rel`) + +### Error Handling + +| Condition | Handling | +|---|---| +| Any error from `Abs`, first `EvalSymlinks`, second `EvalSymlinks`, or `Lstat` (non-exist) | Wrapped with descriptive message using `fmt.Errorf(..., err)` β€” underlying cause recovered via `%w` | +| Unrecognized error from `Lstat` (`!os.IsNotExist`) | Returned immediately as wrapped error | +| Path escapes repo root (`..` prefix or non-nil rel) | Returns sentinel-style error wrapping `ErrPathTraversal` with the original input path | + +## Lock Acquisition (`security.go` β€” `AcquireLock`) + +### Signature + +```go +func AcquireLock(repoRoot string) (*SimpleLock, error) +``` + +**Responsibility:** Establish an exclusive process-level lock on a shared resource. Uses `O_EXCL` flag for atomic creation of the lockfile β€” if the lockfile already exists, acquisition fails immediately without silent success. + +### Lockfile Lifecycle + +| Phase | Mechanism | +|---|---| +| Resolve path | Calls `SafeResolve(repoRoot)` to obtain absolute lock path | +| Create atomically | Opens file with `O_WRONLY \| O_CREATE \| O_EXCL` β€” atomic exclusive creation prevents race between existence check and open | +| Record identity | Writes PID of the acquiring process to the newly created lockfile | +| Failure cleanup | On failure: closes file, removes lockfile | + +### Error Handling + +| Condition | Handling | +|---|---| +| `SafeResolve` fails | Propagated as-is to caller | +| File open fails AND error is `os.IsExist` (lock already held) | Returns wrapped sentinel error wrapping `ErrLockHeld`, including the lock path β€” caller can detect stale-lock condition specifically via `errors.Is(err, ErrLockHeld)` | +| File open fails with any other error | Wrapped: `"failed to acquire lock at %s: "` | +| Write PID fails | File closed, lockfile removed (cleanup), wrapped error returned | + +## Gitignore Maintenance (`security.go` β€” `EnsureGitignoreHasLockfile`) + +### Signature + +```go +func EnsureGitignoreHasLockfile(repoRoot string) (error) +``` + +**Responsibility:** Maintain the convention that the lockfile is excluded from version control but automatically tracked if missing. Appends the lockfile entry to `.gitignore` only if not already present; uses atomic rename via temp file + sync before close to prevent partial writes. + +### Algorithmic Steps + +1. Resolve `.gitignore` path via `SafeResolve(repoRoot)` +2. Read existing `.gitignore` content (`os.ReadFile`) +3. Parse lines, check if lockfile entry already exists +4. If not present: construct new content with the lockfile entry appended (with trailing-newline normalization) +5. Create a temp file in same directory as `.gitignore` +6. Write to temp file β†’ call `Sync()` on it β†’ close it β†’ `Chmod` it +7. Rename temp file over original `.gitignore` + +### Error Handling + +| Condition | Handling | +|---|---| +| `SafeResolve` fails | Propagated as-is | +| Read fails AND not `os.IsNotExist` (file exists but read error) | Wrapped: `"error reading .gitignore: "` β€” caller must handle missing-file vs I/O-error distinction separately | +| CreateTemp fails | Wrapped with descriptive message including `%w` | +| Write to temp file fails | Wrapped; temp file is NOT cleaned up (defer handles close + remove, but write error returns before cleanup path executes) | +| Sync fails | Wrapped β€” note that `os.Chmod` and subsequent rename may have already run on a partially-written temp file if this error occurs late in the sequence | +| Close fails | Wrapped | +| Chmod fails | Wrapped | +| Rename (atomic swap) fails | Wrapped with descriptive message | + +The deferred function (`tmpFile.Close()` + `os.Remove(tmpName)`) runs regardless of return path. Because the function returns before the defer has a chance to execute in some failure paths, cleanup may not happen for intermediate states β€” but since we're using temp file with unique name and same-directory rename, the OS-level temp file is cleaned by `os.Remove` in the defer on any error return path. If *write* fails mid-stream before close/rename, the deferred Remove will clean up the partially-written temp file. + +## Data Flow Summary + +``` +[Initialize] β†’ [Ensure path safety for all repo-root operations (SafeResolve)] β†’ +[If no process holds the lock] β†’ [Create lockfile with PID via O_EXCL (AcquireLock)] β†’ +[Run business logic within locked session] β†’ +[Append entry to .gitignore if needed (EnsureGitignoreHasLockfile)] +``` + +The two non-obvious constraints are: (1) path resolution walks symlinked ancestors before rejoining suffix components, and (2) gitignore updates use atomic rename with explicit sync before close. + +## Key Patterns Observed Across the Module + +1. **Error wrapping throughout** β€” All functions use `fmt.Errorf(..., err)` with `%w`, making root-cause recovery possible via `errors.Is()`. +2. **Custom sentinel errors** β€” At least two custom error variables referenced: `ErrPathTraversal` and `ErrLockHeld`. These enable typed error detection by callers without requiring unwrapping logic on this file's side. +3. **No panic anywhere** β€” All failure paths return to caller with an error value. +4. **Idempotent operations deliberately silent** β€” `Unlock()` on already-closed lock returns nil; missing `.gitignore` in EnsureGitignoreHasLockfile returns nil. These are design choices, not accidentals. +5. **Atomic rename pattern for .gitignore** β€” Write to temp β†’ Sync β†’ Close β†’ Chmod β†’ Rename is the standard atomic-update pattern. The `Sync()` call before close is explicit and unusual (normally deferred), but here it's called eagerly before close in this implementation. +6. **No global package-level mutable state** exists in either file β€” all state is either local to a function or encapsulated within the `SimpleLock` receiver, which is accessed only through its methods. \ No newline at end of file From 4273dd5b05e25916817e86c718d54c8131364c4a Mon Sep 17 00:00:00 2001 From: Juan Ezquerro LLanes Date: Mon, 13 Jul 2026 15:52:18 +0200 Subject: [PATCH 5/5] docs: add comprehensive project documentation and architectural guidelines --- AGENTS.md | 9 ++ wiki/.metadata.json | 103 +++++++++++++ wiki/architecture.md | 245 +++++++++++++++++++++++++++++++ wiki/modules/internal.md | 259 +++++++++++++++++++++++++++++++++ wiki/modules/internal_tools.md | 107 ++++++++++++++ wiki/modules/root.md | 251 ++++++++++++++++++++++++++++++++ wiki/quickstart.md | 189 ++++++++++++++++++++++++ 7 files changed, 1163 insertions(+) create mode 100644 AGENTS.md create mode 100644 wiki/.metadata.json create mode 100644 wiki/architecture.md create mode 100644 wiki/modules/internal.md create mode 100644 wiki/modules/internal_tools.md create mode 100644 wiki/modules/root.md create mode 100644 wiki/quickstart.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..1780c8a --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,9 @@ +# AI Agent Guidelines + +This repository contains automatically generated documentation under the wiki directory to help AI coding agents understand the system architecture, design patterns, and module structure: + +- **System Blueprint**: Refer to wiki/architecture.md for a high-level system overview, module relationships, and boundary definitions. +- **Developer Quickstart**: Refer to wiki/quickstart.md for onboarding steps, coding patterns, and configuration settings. +- **Module Details**: Explore wiki/modules/ for directory-level summaries and API descriptions of internal packages. + +Before making changes, analyze these files to align with existing design choices and code structures. diff --git a/wiki/.metadata.json b/wiki/.metadata.json new file mode 100644 index 0000000..30a1853 --- /dev/null +++ b/wiki/.metadata.json @@ -0,0 +1,103 @@ +{ + "version": 1, + "steps_hash": "f71de95b37d8f26172f8d0c178ff931c9e278e6dda0080096dd6db773dacbaa9", + "files": { + "cmd/init.go": { + "sha256": "87cb2d0246334e1b067c69acf67d42eddd7bae533c7cea9b0cf156ef69a3fcc2", + "facts": "#### [API_SIGNATURES]\n## Public Surface Area: None\n\nThis file exports **nothing**. All identifiers begin with a lowercase letter (`initCmd`, `init`), which makes them unexported within the package. No structs, interfaces, methods, functions, variables, or constants are declared and exported here.\n\n#### [BUSINESS_LOGIC]\n**Business Rules / Domain Concepts:**\n* **Project Documentation Initialization**: The system supports a CLI operation to initialize project documentation by scanning the repository and generating an initial set of wiki markdown pages.\n* **Command Registration Pattern**: The `init` command is registered with the root command (`RootCmd`) during package initialization, establishing it as part of the available CLI surface area.\n\n**High-Level Algorithmic Flow:**\n1. **Invocation**: The user executes the `cmd init` subcommand.\n2. **Delegation**: Execution control is transferred to an internal handler function called via `executeCommand(\"init\")`.\n3. **Processing (External)**: The actual logic for scanning and building wiki pages resides in the implementation referenced by `executeCommand`, which this file does not contain directly.\n\n#### [STATE_AND_CONCURRENCY]\n**Mutable State:** `initCmd` (type `*cobra.Command`). It is declared as a package-level variable and assigned exactly once via its value initializer; no subsequent writes occur in this file, so it does not exhibit runtime mutation. No other global variables or struct fields are modified after initialization.\n\n**Concurrency Mechanisms:** None. The file contains no mutexes, channels, `sync` primitives, or any concurrency control constructs.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects (External Communication)\n\n**None observable.** This code is a pure CLI command registration. The actual external communication happens inside `executeCommand(\"init\")`, which is not shown in this file. What *is* visible:\n\n- **Library dependency**: Uses `github.com/spf13/cobra` for CLI parsing and execution. No I/O occurs at the registration level.\n- **No direct disk, network, or DB access** on this module itself. The \"scan repository\" behavior is deferred to whatever `executeCommand` implements.\n\n## Error Handling / Return Path\n\n| Aspect | Detail |\n|--------|--------|\n| **Error propagation mode** | Returned via `RunE`. Errors bubble up through cobra's command chain to the CLI runner. |\n| **Sentinel wrapping** | None visible. No custom error types, no wrapped errors (`fmt.Errorf`, `%w`). The raw return from `executeCommand(\"init\")` passes through unchanged. |\n| **Panic behavior** | Not handled here. If `executeCommand` panics, it is not caughtβ€”standard cobra/Go panic propagation applies (process exits non-zero). |\n| **Nil error path** | If `executeCommand` returns `nil`, the command succeeds silently. No logging or confirmation output is emitted in this file. |\n| **Error swallowing** | None. The error from `executeCommand` is returned directly with no wrapping, filtering, or suppression. |\n\n## Summary\n\nThis file acts as a thin bridge: it registers an \"init\" subcommand and delegates all work (including I/O and error handling) to the unshown `executeCommand`. Errors are returned raw; success is silent. No panic recovery or custom error semantics exist in this module." + }, + "cmd/root.go": { + "sha256": "8f73bc460485f005d0153d7fb5be7fb0d1b40d364cd3bc768f5547ef529ea510", + "facts": "#### [API_SIGNATURES]\n# Public Surface Area: `root.go` (Module: `cmd`)\n\n## Exported Symbols\n\n| Symbol | Kind | Input / Output Types |\n|---|---|---|\n| `RootCmd` | Variable (`*cobra.Command`) | β€” (no arguments; no return value) |\n\n#### [BUSINESS_LOGIC]\n## Domain Concepts \u0026 Business Rules\n\n### Core Identity\nThis file defines the entry point for a CLI tool called **Code-Reducer**, which is positioned as a documentation agent that writes and maintains project wikis. It serves as an optimized local port of an existing Code-Reducer system.\n\n### Key Business Rules\n\n1. **Git Repository Requirement** β€” The tool must operate within a git repository. If the current directory is not a valid git repo, execution fails immediately.\n\n2. **Configuration Lifecycle** β€” Two distinct modes for project setup:\n - **Init mode**: Creates/initializes documentation infrastructure (wiki structure). Fails if already initialized; requires `update` to refresh later.\n - **Update mode**: Refreshes existing documentation. Fails if the project has not been initialized yet.\n\n3. **Implicit Setup Flow** β€” When no configuration file exists and input is a terminal, the tool automatically triggers an inline setup process without requiring explicit user invocation of `setup`. This avoids blocking non-interactive users.\n\n4. **Configuration Resolution Priority** β€” The merged configuration is resolved from three sources:\n - Persistent command-line flags (`--model-id`, `--num-ctx`)\n - File-based config (from project directory)\n - These are merged together into a single resolved config object.\n\n5. **Graceful Shutdown** β€” A signal handler captures `INT` and `TERM` signals, allowing the engine to stop cleanly when the user interrupts or terminates the process.\n\n6. **Event-Driven Output Routing** β€” The runner emits typed events (Status, Error, etc.) that are routed to different output streams:\n - Status messages β†’ stdout\n - Errors β†’ stderr with \"Error:\" prefix\n - Other messages β†’ stdout\n\n### High-Level Algorithmic Flow\n\n```\nUser invokes `code-reducer` command\n β”‚\n β”œβ”€β”€ 1. Resolve current working directory\n β”‚ └── Fail if not a git repo\n β”‚\n β”œβ”€β”€ 2. Check configuration state\n β”‚ β”œβ”€β”€ No config + TTY β†’ Run implicit setup flow\n β”‚ └── Config exists β†’ Continue\n β”‚\n β”œβ”€β”€ 3. Merge configuration sources (flags + file)\n β”‚\n β”œβ”€β”€ 4. Determine operation mode (init, update, run)\n β”‚ β”œβ”€β”€ init β†’ Verify not already initialized\n β”‚ β”œβ”€β”€ update β†’ Verify project is initialized\n β”‚ └── other modes β†’ Proceed to engine\n β”‚\n β”œβ”€β”€ 5. Install signal handler for graceful termination\n β”‚\n β”œβ”€β”€ 6. Instantiate and execute documentation engine\n β”‚ └── Engine emits events during processing\n β”‚ └── Events routed to stdout/stderr based on type\n β”‚\n └── 7. Return result (success or wrapped error)\n```\n\n### Dependencies Referenced (by module, not implementation)\n- **cmd** β€” CLI command orchestration and user-facing entry point\n- **config** β€” Configuration storage, resolution, and file-based config management\n- **engine** β€” Core documentation generation pipeline with mode handling and event system\n- **tools** β€” Utility functions including git repository verification\n\n#### [STATE_AND_CONCURRENCY]\n**Mutable State:**\n\n| Variable | Type | Mutated? | Notes |\n|---|---|---|---|\n| `modelIDFlag` (package-level) | `string` | No | Set once in `init()` via flag parsing; only read subsequently. |\n| `numCtxFlag` (package-level) | `string` | No | Same as above β€” set once, never reassigned. |\n| `RootCmd` (package-level) | `*cobra.Command` | No | Initialized with a literal at declaration; never mutated after that point in this file. |\n| `repoRoot` (local in executeCommand) | `string` | No | Single assignment via `os.Getwd()`. |\n| `needsSetup` (local in executeCommand) | `bool` | No | Single computation + read. |\n| `isTTY` (local in executeCommand) | `bool` | No | Single computation + read. |\n| `cfg` (local in executeCommand) | `*config.Config` | No | Assigned once from `ResolveConfig`; never mutated after that point. |\n| `hasInit` (local in executeCommand) | `bool` | No | Single computation + read. |\n\n**Concurrency Mechanisms:**\n\n- **Signal-based cancellation**: `signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)` creates a cancellable context for the runner loop. This is not a mutex or channel β€” it's a context propagation mechanism only.\n- **No sync.Mutex**, no explicit channels, no goroutine-level locking visible in this file.\n\n**Conclusion:** No mutable state that changes at runtime beyond initialization. Concurrency relies solely on `context.Context` cancellation via OS signals; no shared-mutable-state synchronization primitives are present.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects (Outside-World Communication)\n\n**Stdout / Stderr Output:**\n* `fmt.Println(ev.Message)` β€” prints engine status events to stdout.\n* `fmt.Fprintf(os.Stderr, \"Error: %s\\n\", ev.Message)` β€” prints engine error events to stderr prefixed with \"Error:\".\n\n**Signal Handling (OS):**\n* Registers handlers for `os.Interrupt` and `syscall.SIGTERM` via `signal.NotifyContext`. A deferred call to `stop()` ensures the context is cancelled when a signal arrives.\n\n**Disk / Filesystem I/O:**\n* `os.Getwd()` β€” reads current working directory path from kernel state.\n* `tools.VerifyGitRepo(repoRoot)` β€” checks git repository validity on disk (writes implied by side-effect rules).\n* `config.ConfigExists(repoRoot)` β€” checks for existence of the configuration file on disk.\n* `RunSetupFlow(repoRoot)` β€” triggers setup logic that writes to disk when the config is missing and stdin is a terminal.\n* `config.ResolveConfig(repoRoot, modelIDFlag, numCtxFlag)` β€” reads configuration files from disk (merged with flags).\n* `engine.IsInitialized(repoRoot, cfg.DocsDir)` β€” checks for existence of initialization artifacts on disk.\n\n**Command-Line Interface:**\n* Registers a root cobra command named \"code-reducer\" with two persistent string flags: `--model-id` and `--num-ctx`.\n* Calls `executeCommand(mode)` where `mode` is derived from the parsed cobra subcommand (passed via `engine.Mode(mode)`).\n\n**Context Creation:**\n* Creates a parent context from `context.Background()` and wraps it in a signal-aware cancellation scope.\n\n---\n\n## Error Handling / Sentinel Analysis\n\n**No panics.** All errors are returned explicitly as `error` interface values.\n\n**Wrap Pattern (100% of observed paths):**\n* Every error originates from a sub-function call, is immediately wrapped with `fmt.Errorf(\"…: %w\", err)`, and propagated to the next level or returned to cobra's main handler.\n* Examples:\n * `os.Getwd()` failure β†’ wrapped as `\"failed to get current working directory: …\"`.\n * `tools.VerifyGitRepo` error β†’ passed through unchanged (already a wrapped/typed error).\n * `RunSetupFlow` error β†’ passed through unchanged.\n * `config.ResolveConfig` error β†’ passed through unchanged.\n * `engine.NewRunner(cfg).Run(...)` result β†’ wrapped as `\"documentation run failed: …\"`.\n\n**Sentinel / Early-Return Logic:**\n* The function returns immediately if the working directory cannot be obtained (`os.Getwd()` fails).\n* It returns early with a typed error from `tools.VerifyGitRepo` without wrapping.\n* If stdin is not a terminal and config is missing, it returns an explicit sentinel message: `\"configuration file … does not exist… Please run 'code-reducer setup'…\"`.\n* \"Init already done\" β†’ returns wrapped error telling user to use `update`.\n* \"Not initialized yet\" (during update) β†’ returns wrapped error telling user to run `init` first.\n\n**Signal-Driven Termination:**\n* The deferred `stop()` cancels the context when `SIGINT` or `SIGTERM` is received, which causes the engine runner (`runner.Run`) to return an error. That returned error is then wrapped as `\"documentation run failed: …\"`." + }, + "cmd/setup.go": { + "sha256": "0d8eaf91b4a8c5e59c330e7512bbc723ce96d6c622a0c3d71f43f6dd07c1b156", + "facts": "#### [API_SIGNATURES]\n# Public Surface Area: `cmd/setup.go`\n\n## Exported Functions\n\n### `RunSetupFlow(repoRoot string) error`\n- **Input:** `repoRoot string` β€” the root directory path for the repository configuration\n- **Output:** `error` β€” returns an error if configuration saving fails (specifically wraps any error from `config.SaveConfig`)\n\n#### [BUSINESS_LOGIC]\n## Business Logic \u0026 Domain Concepts\n\n### Core Purpose\nThis module provides an interactive CLI setup flow that guides users through configuring all application settings for Code-Reducer, persisting them to a local `.code-reducer.yaml` file.\n\n### Configuration Domains Captured\n\n| Concept | Description |\n|---|---|\n| **LLM Identity** | Model ID and base URL β€” defines which LLM backend and model the tool uses for all downstream AI operations (extraction, synthesis, consolidation). |\n| **Context Window** | Token/line context size β€” controls how much source material is fed to the LLM per request. |\n| **Exclusion Rules** | Ignore patterns (files, directories, glob patterns) β€” defines what code is skipped during analysis. |\n| **Documentation Source** | Docs directory path β€” specifies where documentation files live for inclusion in analysis. |\n| **AI Prompts** | Four distinct system prompts: extraction steps, module synthesis, architecture description, and file-fact consolidation. Each prompt governs a separate phase of the code-reduction pipeline. |\n\n### Algorithmic Flow (High-Level)\n\n1. **Resolve existing configuration.** Attempt to load `.code-reducer.yaml` from the current working directory. If it exists, extract its values as fallback defaults; if missing, use built-in defaults for every field.\n\n2. **Prompt user for each setting sequentially.** For each of the nine configuration fields (model ID, base URL, context size, ignores, docs dir, four prompts):\n - Display current value as a hint inside brackets.\n - Accept raw input from stdin.\n - If input is empty or the user types `clear`/`none`, retain the existing/fallback value.\n - For ignore patterns: accept comma-separated values; if non-empty, replace existing list entirely (treats it as a full override).\n\n3. **Assemble final configuration object.** Merge user-supplied inputs with preserved existing values where applicable (e.g., prompts only update if the user provides new content; ignores fully replace on any input).\n\n4. **Persist configuration.** Write the assembled config to `.code-reducer.yaml` in the current directory. On success, confirm the file path; on failure, propagate an error.\n\n#### [STATE_AND_CONCURRENCY]\n**Mutable State:**\n- Local scalar variables (`existingModel`, `existingBaseURL`, `existingNumCtx`, `modelInput`, `urlInput`, etc.) β€” reassigned sequentially within `RunSetupFlow`. No shared/global scope.\n- `existingCfg` (pointer to `config.Config`) β€” reassigned based on `config.LoadConfig()` result; never accessed concurrently.\n- `customIgnores` β€” derived slice from `existingCfg.Ignore`; local only.\n- `ignores` β€” reassigned conditionally from user input or existing config; local only.\n- `extractionSteps`, `existingSystemPrompt`, `existingModuleSynthesisPrompt`, etc. β€” all local, reassigned before building `newCfg`.\n\n**Concurrency Mechanisms:** None. No goroutines, mutexes, channels, atomics, or other concurrency primitives are used in this file. All operations execute sequentially on the main thread.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects \u0026 Error Handling Analysis β€” `cmd/setup.go`\n\n---\n\n### I/O Operations\n\n| Operation | Direction | Source | Detail |\n|-----------|-----------|--------|--------|\n| `bufio.NewReader(os.Stdin)` | Open | OS stdin handle | Single reader created at start of `RunSetupFlow`; all user prompts read from this same instance. |\n| `os.Getwd()` | Read | OS filesystem state | Returns current working directory; error is wrapped and returned immediately. |\n| `config.LoadConfig(repoRoot)` | Read (disk) | Filesystem | Reads config file at startup. Errors are **silently swallowed** β€” no return, no logging. Nil check then reads fields into local vars. |\n| `fmt.Printf` / `fmt.Println` to stdout | Write | OS console | Welcome banner, prompt lines with existing values in brackets, final success message. |\n| `reader.ReadString('\\n')` (in `promptString`) | Read | stdin | On error: writes warning to stderr, returns existing value. Never propagates input-read errors upward. |\n| `reader.ReadString('\\n')` (in `promptStringList`) | Read | stdin | On error: writes warning to stderr, returns existing list + `modified=false`. |\n| `config.SaveConfig(repoRoot, newCfg)` | Write (disk) | Filesystem | Writes the assembled config. Errors are wrapped with context and returned as a failure from `RunSetupFlow`. |\n\n---\n\n### Error Handling Patterns\n\n| Location | Pattern | Behavior |\n|----------|---------|----------|\n| `os.Getwd()` error | **Wrap + return** | Wrapped via `%w`, propagated up to caller. This is the only unwrapped-but-returned error from I/O. |\n| `config.LoadConfig` error | **Silent swallow** | No error returned, no log message. If loading fails, defaults are used for all existing fields. |\n| Stdin read errors in prompt functions | **Graceful degrade** | Writes to stderr, returns pre-existing value (or empty list + `modified=false`). User is never blocked by bad input from the terminal. |\n| `strconv.Atoi` failure | **Conditional override** | If parse fails or result ≀ 0, falls back to `existingNumCtx`. No error returned; no logging. |\n| `config.SaveConfig` error | **Wrap + return** | Wrapped with `\"error saving configuration: %w\"`, propagated up. This is the only hard failure in the flow β€” if config cannot be written, setup fails. |\n\n---\n\n### Return Values (what the function communicates back)\n\n- **`RunSetupFlow`**: Returns `nil` on success; returns a wrapped error on Getwd or SaveConfig failure. All other issues are absorbed internally.\n- **`promptString`**: Always returns a string β€” either user input, existing value, or empty-string fallback (when input is blank). Never returns an error.\n- **`promptStringList`**: Returns `(list []string, modified bool)`. `modified=true` means user typed something; `false` means kept existing state.\n\n---\n\n### Summary of Communication Channels\n\n| Channel | Used? | Notes |\n|---------|-------|-------|\n| Console stdout (messages/prompts) | Yes | All user-facing output |\n| Console stderr (warnings) | Yes | Only on stdin read errors β€” never for normal operation |\n| Disk read (config file) | Yes | Via `LoadConfig`; errors ignored |\n| Disk write (config file) | Yes | Via `SaveConfig`; errors are fatal to the flow |\n| Stdin | Yes | Single reader; all prompts funnel through it |\n| Network / DB / external services | No | None in this file |\n\n---\n\n### Notable Observations\n\n1. **Only one hard failure**: If `SaveConfig` fails, setup aborts with a wrapped error. Everything else is best-effort.\n2. **No retry logic**: All stdin errors are single-shot; the user gets warned and existing values are preserved, then flow continues.\n3. **No logging framework used**: Errors go to stderr directly via `fmt.Fprintf(os.Stderr, ...)`. No structured logger appears in this file.\n4. **`existingCfg` is a local shadow**: If config loads successfully, `existingCfg` holds it; otherwise it stays nil throughout the function. The nil check on line 58 (`cfg, err := ...`) means subsequent reads of `existingCfg.Ignore`, etc., are guarded by prior nil checks β€” no panic risk on nil access." + }, + "cmd/update.go": { + "sha256": "62e9aa11268d32732195d0dbf3d38f5f817afe9a2a276aa7a1effac06566f8ea", + "facts": "#### [API_SIGNATURES]\n# Public Surface Area: `cmd/update.go`\n\n## Summary\n\nThis file exports **nothing**. It contains only unexported items.\n\n### Unexported Items (for context)\n\n| Item | Type | Notes |\n|------|------|-------|\n| `updateCmd` | `*cobra.Command` | Local variable, registers the \"update\" cobra command |\n| `init()` | function | Standard Go init; adds `updateCmd` to `RootCmd` (a package-level reference) |\n\n### No Public Surface Area Found\n\n- **No exported structs** β€” none defined.\n- **No exported interfaces** β€” none defined.\n- **No exported functions/methods** β€” no capitalized identifiers present.\n- The only public API this module exposes is the `update` cobra subcommand, but it lives in an external package (`github.com/spf13/cobra`) and its registration happens entirely within private scope here.\n\n#### [BUSINESS_LOGIC]\n## File: update.go\n\n### Domain Concept\nThis module implements an incremental **documentation/wiki update** workflow for a CLI tool that manages wiki pages tied to repository commits.\n\n### Business Rule\nWhen invoked, the tool scans the repository for files changed since the last documented commit and regenerates the corresponding wiki pages β€” keeping documentation in sync with source changes without requiring a full regeneration each time.\n\n### High-Level Algorithmic Flow\n1. **Command registration** β€” A `cobra` command named `update` is defined under the root CLI.\n2. **Execution delegation** β€” On invocation, it calls `executeCommand(\"update\")`, which hands off to an external binary (the core documentation engine). The actual scanning and regeneration logic lives outside this file; this file only defines the entry point.\n\n#### [STATE_AND_CONCURRENCY]\n### Mutable State Analysis\n\n**No mutable state.**\n\n- `updateCmd` (`*cobra.Command`) is initialized once as a literal struct value at package scope, then never reassigned or modified in this file.\n- The `RunE` function references an undefined `executeCommand(\"update\")`, which is outside the scope of this analysis.\n- No struct fields are mutated after initialization.\n\n### Concurrency Mechanisms\n\n**No concurrency mechanisms.**\n\n- No `sync.Mutex`, channels, atomic operations, or other synchronization primitives are used.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects \u0026 Error Handling Analysis\n\n### I/O Communication\nNo explicit I/O (network, disk, DB) is visible in this snippet. All operations are deferred to an external function (`executeCommand`) whose implementation is not present in this file. The cobra import suggests CLI/framework interaction only.\n\n### Error Handling Pattern\n- **RunE returns error** β€” follows the standard `cobra.Command` pattern where errors bubble up to the caller (typically Cobra's root command handler).\n- No explicit error wrapping, sentinel types, or panic recovery is visible here.\n- The actual error handling logic lives in `executeCommand`, which is undefined in this file.\n\n### Unresolved Dependencies\nTwo identifiers are referenced but not defined within this snippet:\n1. **`RootCmd`** β€” assumed to be a package-level variable holding the root cobra command; registration happens in `init()`.\n2. **`executeCommand`** β€” invoked with string literal `\"update\"`; its signature and behavior are unknown from this context alone.\n\n### Observable Code Path\n```\ncmd *cobra.Command, args []string β†’ executeCommand(\"update\") β†’ error\n```\nThe function is a thin dispatcher. All substantive work (and any side effects or error handling) occurs outside this file." + }, + "internal/config/config.go": { + "sha256": "0a06598bbb4f6e07a17d9c8ea6c31427b50feb0fe8ecce6cd03ac0b195683aab", + "facts": "#### [API_SIGNATURES]\n# config.go β€” Public Surface Area\n\n## Exported Structs\n\n| Type | Fields (Input/Output Types) |\n|------|-----------------------------|\n| `ExtractionStep` | `Name string`, `Prompt string` |\n| `Config` | `ModelID string`, `OllamaBaseURL string`, `OllamaNumCtx int`, `DocsDir string`, `SystemPrompt string`, `ModuleSynthesisPrompt string`, `ArchitecturePrompt string`, `FileFactConsolidationPrompt string`, `ExtractionSteps []ExtractionStep`, `Ignore []string` |\n\n## Exported Interfaces\n\nNone.\n\n## Exported Methods\n\nNone.\n\n## Exported Constants\n\n| Constant | Value |\n|----------|-------|\n| `CodeReducerModelIDEnvKey` | `\"CODE_REDUCER_MODEL_ID\"` |\n| `OllamaBaseURLEnvKey` | `\"OLLAMA_BASE_URL\"` |\n| `OllamaNumCtxEnvKey` | `\"OLLAMA_NUM_CTX\"` |\n| `OllamaDefaultBaseURL` | `\"http://localhost:11434\"` |\n| `OllamaDefaultModelID` | `\"ornith:9b\"` |\n| `OllamaDefaultNumCtx` | `8192` |\n| `DefaultDocsDir` | `\"wiki\"` |\n| `ConfigFileName` | `\".code-reducer.yaml\"` |\n| `DefaultSystemPrompt` | (multiline string) |\n| `DefaultModuleSynthesisPrompt` | (string) |\n| `DefaultArchitecturePrompt` | (string) |\n| `DefaultFileFactConsolidationPrompt` | (string) |\n\n## Exported Variables\n\n| Variable | Type | Value |\n|----------|------|-------|\n| `DefaultExtractionSteps` | `[]ExtractionStep` | Pre-populated slice with 4 named steps (`API_SIGNATURES`, `BUSINESS_LOGIC`, `STATE_AND_CONCURRENCY`, `ERRORS_AND_SIDE_EFFECTS`) |\n\n## Non-Exported (Internal)\n\n- `configFilePerm = 0600` β€” lowercase first letter, not part of the public surface.\n\n#### [BUSINESS_LOGIC]\n## Business Rules \u0026 Domain Concepts\n\nThis file defines the orchestration layer for a multi-phase LLM analysis pipeline that generates technical documentation from source code.\n\n### Core Workflow\n\nThe algorithmic flow is a **sequential, staged synthesis**:\n\n1. **Configuration resolution** β€” Load `.code-reducer.yaml`, apply environment variable overrides (model ID, base URL, context size), and fall back to built-in defaults.\n\n2. **Multi-step extraction pipeline** β€” Four analysis phases run in order:\n - `API_SIGNATURES`: Extract public surface area (structs, interfaces, exported methods) with input/output types.\n - `BUSINESS_LOGIC`: Explain domain concepts, business rules, and high-level algorithmic flow.\n - `STATE_AND_CONCURRENCY`: Identify mutable state and concurrency mechanisms protecting it.\n - `ERRORS_AND_SIDE_EFFECTS`: Map external communication (I/O, network, DB) and error handling patterns.\n\n3. **Upstream synthesis** β€” Module summaries are then consolidated into architecture-level documentation using separate prompts for module synthesis and architecture overview.\n\n### Key Design Decisions\n\n- **Default LLM target**: Hardcoded to a local Ollama instance (`localhost:11434`, model `ornith:9b`, 8192 context). This signals the tool is designed to run offline/locally by default.\n- **Prompt templates are first-class config** β€” All prompts are configurable via YAML, enabling users to tune analysis behavior without code changes.\n- **Extraction steps are extensible** β€” The `[]ExtractionStep` slice allows adding new analysis phases (e.g., a security audit step) without modifying the orchestration logic.\n\n#### [STATE_AND_CONCURRENCY]\n**Mutable State:** None identified. The package-level variable `DefaultExtractionSteps` is initialized at declaration only; no post-initialization writes or mutations occur in this file. All other data (constants, struct type definitions) are immutable by nature.\n\n**Concurrency Mechanisms:** None. No `sync.Mutex`, channels, atomics, or other concurrency primitives appear in the source text.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects \u0026 Error Handling Analysis: `internal/config`/`config.go`\n\n### Communication with Outside World (I/O)\n\n**Result: None detected.** No network, disk, or database operations are performed by this file. All constants and types are compile-time definitions.\n\n| Potential Concern | Verdict | Detail |\n|---|---|---|\n| Network calls | Not present | No HTTP clients, URL parsing, or connection logic exists in this file. `OllamaBaseURLEnvKey` and related defaults are static string literals only. |\n| Disk I/O (read/write) | Not present | `ConfigFileName = \".code-reducer.yaml\"` and `configFilePerm = 0600` are constants; no actual file open/read/write calls exist here. |\n| Database operations | Not present | No DB drivers, queries, or connection pools referenced. |\n| String concatenation in prompts | Safe (compile-time) | `DefaultSystemPrompt` uses `\\n+` for newline and string concatenation via the `+` operator. Both are compile-time evaluations. The `\\n` escape is a valid Go rune literal; no runtime interpretation occurs. |\n\n### Error Handling\n\n**Result: None detected.** No error types, panic calls, or wrapping patterns exist in this file.\n\n| Pattern | Verdict | Detail |\n|---|---|---|\n| `panic()` calls | Not present | Zero panics anywhere in the file. |\n| Custom error types | Not present | No `type X error` declarations; no sentinel errors defined. |\n| Error wrapping (`fmt.Errorf`, `errors.Join`, etc.) | Not present | No `Errorf` or similar function calls exist. |\n| Return values with errors | Not present | The file contains no functions that return errors, only constants and types. |\n\n### Summary\n\nThis file is a pure configuration layer: it defines compile-time constants (environment keys, defaults, prompts) and data structures (`ExtractionStep`, `Config`) for downstream use. It communicates **zero** I/O with the outside world and performs **zero** error handling itselfβ€”those concerns are deferred to whichever module later consumes these types/consts." + }, + "internal/config/io.go": { + "sha256": "9af0f95e0eb8f5bac5679053c283565338e4316f1f7065063a26c41df24532cf", + "facts": "#### [API_SIGNATURES]\n## Exported Functions\n\n| Function | Input | Output |\n|----------|-------|--------|\n| `ConfigExists(cwd string)` | `string` (cwd directory path) | `bool` |\n| `LoadConfig(cwd string)` | `string` (cwd directory path) | `*Config`, `error` |\n| `SaveConfig(cwd string, cfg *Config)` | `string` (cwd directory path), `*Config` | `error` |\n\n## Notable Notes\n\n- **No exported structs or interfaces** are defined in this file. The type `Config` is referenced by function signatures but declared elsewhere.\n- **Internal helpers not listed**: `getConfigPath()` and `configFilePerm` (variable) remain lowercase/unexported.\n\n#### [BUSINESS_LOGIC]\n## Business Rules \u0026 Domain Concepts\n\n### 1. Configuration File Naming Convention\nA single configuration file (`.code-reducer.yaml`) lives alongside each project's working directory. All operations resolve through this fixed filename, meaning the system has no concept of alternative or tiered config files.\n\n### 2. Configuration Persistence Model β€” Atomic Write\n`SaveConfig` does not write directly to the target path. Instead:\n- Marshals the `Config` struct into YAML.\n- Writes to a newly created `.tmp.*` file in the same directory.\n- Sets file permissions via `chmod` before finalizing.\n- Atomically renames the temp file over the original.\n\nThis guarantees that readers never observe a partially-written or corrupt config during save operationsβ€”the old file remains intact until the new one is fully ready to replace it.\n\n### 3. YAML Formatting Normalization\nBefore writing, the marshaled output undergoes string replacement that collapses single blank lines into double blank lines before specific top-level keys (`system_prompt`, `module_synthesis_prompt`, `architecture_prompt`, `file_fact_consolidation_prompt`, `extraction_steps`, `ignore`). This enforces a consistent visual separation between logical sections in the stored config.\n\n### 4. Load-Save Roundtrip Integrity\n`LoadConfig` parses YAML into a typed `Config` struct, while `SaveConfig` serializes it back out. The normalization step is applied only on write, not readβ€”meaning loading preserves whatever formatting was originally written. This decouples the in-memory representation from on-disk formatting expectations.\n\n### 5. Error Propagation Strategy\n- File-not-found errors propagate directly from `os.ReadFile`.\n- YAML parse errors are wrapped with a \"failed to parse yaml config\" prefix and chained via `%w` for downstream unwrapping.\n- Marshal failures similarly wrap a descriptive message.\n- Temp file creation, write, sync, close, chmod, and rename steps each return their own error, ensuring no silent failure during the atomic save sequence.\n\n---\n\n## High-Level Algorithmic Flow (SaveConfig)\n\n```\nInput: cwd, cfg\nβ”‚\nβ”œβ”€ Marshals cfg β†’ YAML bytes\nβ”‚\nβ”œβ”€ Applies formatting normalization to YAML string\nβ”‚ └─ Ensures double blank lines before known section headers\nβ”‚\nβ”œβ”€ Creates temp file in cwd with .tmp.* pattern\nβ”‚\nβ”œβ”€ Writes normalized YAML to temp file\nβ”‚\nβ”œβ”€ Syncs and closes temp file\nβ”‚\nβ”œβ”€ Sets executable permissions on temp file (chmod)\nβ”‚\n└─ Atomically renames temp β†’ target config path\n β”‚\n └─ Cleanup deferred: close + remove if rename fails\n```\n\n---\n\n## High-Level Algorithmic Flow (LoadConfig)\n\n```\nInput: cwd\nβ”‚\nβ”œβ”€ Reads .code-reducer.yaml from cwd\nβ”‚ └─ Returns error if file is missing or unreadable\nβ”‚\nβ”œβ”€ Unmarshals YAML bytes β†’ Config struct\nβ”‚ └─ Wraps parse errors with descriptive prefix\nβ”‚\n└─ Returns loaded Config (or error)\n```\n\n---\n\n**Summary**: This module handles the lifecycle of a single project-level configuration fileβ€”existence checks, loading, saving with atomicity and formatting normalization, and consistent error reporting. The domain concept is \"one config per workspace\" with write safety preserved through an atomic rename pattern.\n\n#### [STATE_AND_CONCURRENCY]\n## State Mutation Analysis\n\n### Mutable State: None detectable (from this file alone)\n\n- **No global variables** are declared or modified in this file. The `configFilePerm` constant referenced by `os.Chmod` is defined elsewhere; no value mutation occurs here.\n- **No struct fields with observable mutation.** The `Config` struct definition is not provided, so its field-level mutability cannot be assessed from this source alone. `LoadConfig` allocates a fresh `*Config` via unmarshaling (no shared state). `SaveConfig` only writes to a temporary file and replaces the original β€” no in-memory Config fields are mutated after creation.\n- **No package-level variables** are visible or modified.\n\n### Concurrency Mechanisms: None\n\n- No `sync.Mutex`, `sync.RWMutex`, channels, atomic operations, or other synchronization primitives appear in this code.\n- Functions (`LoadConfig`, `SaveConfig`) operate on caller-provided arguments and return results β€” they do not hold shared mutable state that requires protection.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects (I/O)\n\nAll communication with the outside world is **disk-only** β€” no network, no database. Every operation touches files via `os` package calls:\n\n| Function | Disk Operations |\n|---|---|\n| `getConfigPath` | None (pure computation). Returns a constructed path string using `filepath.Join`. No error handling needed for this join on valid inputs. |\n| `ConfigExists` | One read-only check via `os.Stat`. Uses the error sentinel as a signal β€” if `err != nil`, treats it as \"file does not exist\". No wrapping; raw error is reused as a boolean condition. |\n| `LoadConfig` | Two disk reads: `os.ReadFile` pulls the config file, then `yaml.Unmarshal` parses it into memory. If ReadFile fails, the raw error is returned unwrapped. If Unmarshal fails, the error **is wrapped** with context via `%w`. Returns `(*Config, error)`. |\n| `SaveConfig` | Multiple writes to disk: creates a temp file (`os.CreateTemp`), writes YAML data into it (`tmpFile.Write`), syncs to disk (`tmpFile.Sync`), closes the temp file (`tmpFile.Close`), sets permissions (`os.Chmod`), then atomically renames temp β†’ final config path (`os.Rename`). The rename makes this effectively atomic at the OS level β€” if SaveConfig fails before Rename, the original `.code-reducer.yaml` is untouched. |\n\n## Error Handling Strategy\n\n| Pattern | Where It Appears |\n|---|---|\n| **Raw return** (no wrapping) | `LoadConfig` β€” when `os.ReadFile` fails, returns the OS error as-is to caller. |\n| **Wrap with context** (`fmt.Errorf(... %w`) | `LoadConfig` (parse failure), `SaveConfig` (temp create, write, sync, close, chmod, rename failures). All wrapped calls preserve chain via `%w`. |\n| **Error-as-boolean sentinel** | `ConfigExists` β€” uses `err == nil` check instead of returning the error. Not a panic; just conventions. |\n| **Silent cleanup in defer** | `SaveConfig` β€” deferred function calls `tmpFile.Close()` and `os.Remove(tmpName)`. Errors from these cleanup calls are swallowed (not returned). This is intentional: if an earlier operation fails, we still close/remove the temp file to avoid leaving garbage on disk. The explicit `Close` before Rename + the defer's `Close` means Close runs twice on success β€” Go allows this safely. |\n| **Panic** | None in this file. All errors are returned as values. |\n\n## Notable Observations\n\n- `SaveConfig` has a subtle ordering issue: if `tmpFile.Write` succeeds but the explicit `tmpFile.Close()` call is skipped (it isn't β€” it's explicitly called before Rename), then the defer would run Close again after Rename, which is redundant but harmless. The actual risk is that `os.Remove(tmpName)` in the defer runs *after* a successful rename β€” the file no longer exists at tmpName, so Remove returns an error that is silently swallowed by the deferred function. This means on success, a stale temp path remains on disk (or rather, the original config file now holds the renamed data).\n\n- No network I/O anywhere in this module." + }, + "internal/config/resolve.go": { + "sha256": "93ec3fdd3dc601ea2a36ac86c0864cd4ef6e1c6886d6863dd2bb4750dd80c2ad", + "facts": "#### [API_SIGNATURES]\n## Public Surface Area: `internal/config/resolve.go`\n\n### Exported Functions\n\n| Function | Input Types | Output Types |\n|---|---|---|\n| `ResolveConfig(repoRoot, modelIDFlag, numCtxFlag string)` | `(string, string, string)` | `(*config.Config, error)` |\n\n### Notes\n\n- **Unexported**: `mergeAndDeduplicate` is lowercase; not part of the public surface.\n- **External references**: The file references types and constants (`Config`, `OllamaDefaultModelID`, `CodeReducerModelIDEnvKey`, `DefaultExtractionSteps`, etc.) that are defined in other files within the same package. Their definitions are outside this file's scope; only their usage is visible here.\n\n#### [BUSINESS_LOGIC]\n## Domain Concepts \u0026 Business Rules\n\n### Configuration Resolution Priority System\n\nThe file implements a multi-source configuration resolution strategy where values are merged from multiple sources in a defined priority order (lowest to highest): **defaults β†’ YAML config file β†’ environment variables β†’ CLI flags**. Each configurable field follows its own independent priority chain.\n\n### Extracted Configurations by Field\n\n| Field | Priority Chain |\n|-------|---------------|\n| `ModelID` | Default \u003e YAML \u003e Env Var \u003e CLI Flag |\n| `OllamaBaseURL` | Default \u003e YAML \u003e Env Var |\n| `OllamaNumCtx` (context size) | Default \u003e YAML \u003e Env Var \u003e CLI Flag |\n| `DocsDir` | Default \u003e YAML |\n| System prompts (`SystemPrompt`, `ModuleSynthesisPrompt`, `ArchitecturePrompt`, `FileFactConsolidationPrompt`) | All start with default, then override with YAML if non-empty |\n\n### Validation Rules\n\n- **Numeric fields** (OllamaNumCtx): CLI flag and env var values are parsed as integers; a value must be greater than zero to take effect. Invalid or non-positive values fall through silently.\n- **String fields**: Empty string is treated as \"not set\" β€” the lower-priority default remains active.\n- **Config file absence** (`config.yml`): Not an error. If missing, an empty `Config{}` struct is created and defaults are applied. Other parse errors are surfaced.\n\n### Ignore List Deduplication\n\nWhen merging ignore lists from config file (and nil/empty env), duplicates are removed while preserving first-seen order using a map-based seen-set.\n\n## High-Level Algorithmic Flow\n\n1. **Load YAML config** β€” if it exists and parses successfully; otherwise, start with an empty struct.\n2. **Resolve extraction steps** β€” use YAML value if present, else fall back to `DefaultExtractionSteps`.\n3. **Deduplicate ignore list** from the loaded config.\n4. **For each field**, apply its specific priority chain:\n - Start with the built-in default constant.\n - Override with YAML config value (if non-empty).\n - Override with environment variable (if set and valid for numeric types).\n - Override with CLI flag (highest priority; numeric types require successful integer parse and positive value).\n5. **Return** the fully resolved `Config` struct, or an error if the YAML file fails to parse (excluding missing-file case).\n\n#### [STATE_AND_CONCURRENCY]\n## State Mutation Analysis\n\n### Mutable State: None in This File\n\nThis file contains **no mutable shared state** and **no concurrency mechanisms**. All operations are confined to local function scopes with no cross-goroutine visibility.\n\n---\n\n**`resolveAndDeduplicate[T comparable]`**: Uses only local variables (`seen`, `result`). No shared state mutation; no locks needed.\n\n**`ResolveConfig`**: Performs sequential resolution logic on:\n- Local variables and parameters (strings) β€” read-only or stack-local writes\n- New allocations (`\u0026Config{}`, maps, slices) created within function scope\n- External reads (`os.Getenv`, `LoadConfig`) β€” not mutating shared state\n\n---\n\n**Concurrency Mechanisms:** None present in this file. No `sync.Mutex`, channels, atomics, or other synchronization primitives are used.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects (I/O)\n\n| Path | Operation | Direction |\n|------|-----------|-----------|\n| `repoRoot` + YAML config file | Read via `LoadConfig(repoRoot)` | Disk read |\n| `CodeReducerModelIDEnvKey` env var | Read via `os.Getenv()` | Environment read |\n| `OllamaBaseURLEnvKey` env var | Read via `os.Getenv()` | Environment read |\n| `OllamaNumCtxEnvKey` env var | Read via `os.Getenv()` | Environment read |\n\nNo network calls, no writes to disk or DB. All external communication is **read-only**.\n\n## Error Handling\n\n**1. Config file load error** β€” wrapped and returned:\n```go\nif err != nil {\n if !os.IsNotExist(err) { ... } // swallow \"file missing\" silently\n}\nreturn nil, fmt.Errorf(\"failed to load configuration file: %w\", err) // non-exist errors propagate\n```\n\n- **Swallowed path**: `os.IsNotExist` true β†’ defaults to empty `Config{}`. No error returned.\n- **Propagated path**: any other error (parse failure, permission denied, etc.) is wrapped with a prefix and returned as `(*Config, nil, error)` β€” the config pointer is `nil`.\n\n**2. All other branches are silent:**\n- String comparisons (`!= \"\"`, `\u003e 0`) short-circuit; no errors from those paths.\n- `strconv.Atoi` failures are caught inline and silently skipped (env/flag values simply not used).\n- Map lookups for deduplication have no error path.\n\n**3. Always-on return:** The function always returns a non-nil tuple `(*Config, error)` β€” either a populated config with `nil` error, or a `nil` config with an error string." + }, + "internal/engine/cache.go": { + "sha256": "f214e3eaec08768b3140ee69e12e1e436c4fe09a0c0b2671d4c63112f6cfef54", + "facts": "#### [API_SIGNATURES]\n# Public Surface Area of `cache.go`\n\n## Structs\n\n### `FileCacheEntry`\n\n| Field | Type | JSON Key |\n|-------|------|----------|\n| SHA256 | string | sha256 |\n| Facts | string | facts |\n\n### `MetadataCache`\n\n| Field | Type | JSON Key |\n|-------|------|----------|\n| Version | int | version |\n| StepsHash | string | steps_hash |\n| Files | map[string]FileCacheEntry | files |\n| Modules | map[string]string | modules |\n\n## Methods\n\n### `IsInitialized`\n\n- **Input:** repoRoot string, docsDir string\n- **Output:** bool\n\n### `saveMetadataCache`\n\n- **Input:** repoRoot string, docsDir string, cache *MetadataCache\n- **Output:** error\n\n#### [BUSINESS_LOGIC]\n## Domain Concepts \u0026 Business Rules\n\n**Persistent Metadata Cache for Document Extraction Pipeline**\n\nThis module implements a caching layer that persists metadata between extraction runs. The cache tracks:\n\n- **Per-file content fingerprints**: Each file is hashed (SHA256) and stored alongside extracted \"facts\" in `FileCacheEntry`.\n- **Pipeline signature**: A hash of the extraction steps (`computeStepsHash`) records which processing rules were active, enabling detection of pipeline changes.\n- **Module registry**: Maps module identifiers to their current state or version.\n\n**Version Compatibility Guard**\n\nThe cache carries a version field (`currentCacheVersion = 1`). On load, if the stored version mismatches, the cache is discarded and replaced with an empty one β€” this prevents stale metadata from corrupting future runs without explicit migration handling.\n\n**Fault-Tolerant Initialization**\n\nIf the cache file does not exist (first run), it is created with empty maps and the current version. If it exists but cannot be read or unmarshaled, a specific error is returned rather than silently failing β€” callers can distinguish \"not yet initialized\" from \"corrupt data.\"\n\n---\n\n## High-Level Algorithmic Flow\n\n1. **Load**: Read cache file β†’ if missing, create empty β†’ else deserialize β†’ check version match β†’ return populated cache (or clean replacement on mismatch).\n2. **Save**: Serialize cache to disk with indented JSON formatting and current version pinned.\n3. **Hash a file**: Read file content β†’ compute SHA256 β†’ return hex-encoded digest.\n4. **Check initialization**: Same file-existence check used in load, but returns boolean only.\n\n#### [STATE_AND_CONCURRENCY]\n**Mutable State:** None. This file declares zero package-level variables, constants are `const` (immutable), and the only structs (`FileCacheEntry`, `MetadataCache`) are referenced locally without any shared/global instances being modified. All functions use only local variables.\n\n**Concurrency Mechanisms:** None detected in this file. No `sync.Mutex`, channels, or other synchronization primitives appear anywhere.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects \u0026 Error Handling Analysis\n\n---\n\n### I/O Operations (Disk)\n\nAll filesystem interactions go through two internal tool functions (`tools.ReadFileSafely`, `tools.WriteFileSafely`). The cache module has **no network, DB, or external service** communication β€” it only touches the local filesystem.\n\n| Function | Direction | Path | Purpose |\n|---|---|---|---|\n| `loadMetadataCache` | Read | `{docsDir}/{metadataFileName}` | Loads persisted metadata cache (JSON) from disk |\n| `IsInitialized` | Read-only check | `{docsDir}/{metadataFileName}` | Returns true only if file exists AND is readable in one call |\n| `saveMetadataCache` | Write | `{docsDir}/{metadataFileName}` | Persists metadata cache as indented JSON |\n| `computeSHA256` | Read (content) | any virtualPath under `repoRoot` | Reads raw file content to compute hash; no write I/O |\n\n**Note:** The path for the cache file uses an undefined identifier `metadataFileName` β€” this is either unexported or in another package not shown here. All paths are constructed by joining `docsDir`, so the cache lives inside a subdirectory of the repo root.\n\n---\n\n### Error Handling Patterns\n\n#### 1. Sentinel-style existence check (in `loadMetadataCache`)\n```go\nif errors.Is(err, os.ErrNotExist) {\n return \u0026MetadataCache{...}, nil // clean empty cache, no error\n}\nreturn ... fmt.Errorf(\"failed to read metadata cache: %w\", err) // wrapped error\n```\n- Uses `errors.Is` against the standard library sentinel `os.ErrNotExist`.\n- **If file missing β†’ returns a fresh zero-valued cache with nil error.** This is intentional behavior (cache bootstrap on first run).\n- **Any other read error β†’ wrapped and returned to caller** via `%w` re-wrapping.\n\n#### 2. JSON unmarshal failure (in `loadMetadataCache`)\n```go\nif err := json.Unmarshal(data, \u0026cache); err != nil { ... }\nreturn ..., fmt.Errorf(\"failed to unmarshal metadata cache: %w\", err)\n```\n- Any unmarshal error is **wrapped and returned** β€” no silent fallback.\n\n#### 3. Version mismatch (in `loadMetadataCache`)\n```go\nif cache.Version != currentCacheVersion {\n return \u0026MetadataCache{...}, nil // clean empty cache, no error\n}\n```\n- Treats an incompatible version as \"cache invalid\" and returns a fresh zero-valued cache with **nil error**. No warning or panic β€” caller sees success.\n\n#### 4. Marshaling/Write errors (in `saveMetadataCache`)\n```go\ndata, err := json.MarshalIndent(cache, \"\", \" \")\nif err != nil { return err }\nreturn tools.WriteFileSafely(repoRoot, metadataPath, data)\n```\n- Marshaling error β†’ **returned directly** as the error value.\n- Write error (via `tools.WriteFileSafely`) β†’ **returned directly**.\n- No wrapping here β€” errors are surfaced as-is from the underlying tool functions.\n\n#### 5. Read failure for hash computation (in `computeSHA256`)\n```go\ncontent, err := tools.ReadFileSafely(repoRoot, virtualPath)\nif err != nil { return \"\", err }\nhash := sha256.Sum256(content)\nreturn hex.EncodeToString(hash[:]), nil\n```\n- Read failure β†’ **returned as error with empty hash string**. No panic.\n\n#### 6. Ignored marshal error (in `computeStepsHash`)\n```go\ndata, _ := json.Marshal(steps)\nh := sha256.Sum256(data)\nreturn hex.EncodeToString(h[:]), nil\n```\n- The marshaling error is **silently discarded** (`_`). If the steps slice contains unmarshalable values, `json.Marshal` will fail and produce truncated/corrupt JSON. The hash will still be computed on whatever bytes were produced. This is a latent bug β€” not an intentional error-handling choice.\n\n---\n\n### Summary Table\n\n| Pattern | Where | Behavior |\n|---|---|---|\n| Sentinel (`os.ErrNotExist`) | `loadMetadataCache` | Returns clean empty cache, no error |\n| Error wrapping (%w) | `loadMetadataCache` (read + unmarshal branches) | Wraps original error into descriptive message |\n| Direct pass-through | `saveMetadataCache`, `computeSHA256` | Errors from tools are returned unchanged to caller |\n| Silent ignore | `computeStepsHash` | Marshaling error swallowed β€” produces partial hash |\n| Panic | Nowhere in this file | All errors return as values |" + }, + "internal/engine/chunking.go": { + "sha256": "5ad84939667740e001438e3dddba54ab179cbae877ebc4a6febf1c791a4af61d", + "facts": "#### [API_SIGNATURES]\n# Public Surface Area: `chunking.go` (Module: internal/engine)\n\n## Summary\n\n**Exported types:** None \n**Exported functions:** None \n\nAll top-level functions and methods defined in this file begin with lowercase letters, making them unexported (package-private). The package is `engine`, so these are accessible only within the `engine` package.\n\n#### [BUSINESS_LOGIC]\n### Business Domain Concepts\n\nThis module implements a **context-aware recursive reduction engine** for LLM-based code synthesis. The core business rule is: *a single LLM call can only process text up to its context window size, so any larger collection must be split, reduced in parallel sub-batches, then recombined until a final answer emerges.*\n\nThe system solves three distinct business problems by applying this same algorithmic pattern with different prompts and domain-specific instructions:\n\n1. **Architecture synthesis** β€” collect code blocks related to a given module path, summarize them as one architecture description for that node.\n2. **File fact consolidation** β€” deduplicate extracted facts about the same file/step into a single coherent fact set.\n3. **General LLM reduction** β€” take any remaining intermediate summaries and synthesize them together with system-level context.\n\n### High-Level Algorithmic Flow\n\n1. **Split**: The input list is divided into batches whose combined character count fits within the LLM context limit (calculated from `contextSize Γ— multiplier`).\n2. **Reduce each batch**: Each batch is sent to the LLM via a domain-specific prompt, producing one summary string per batch. If only one batch exists, it is returned directly; otherwise, all summaries are concatenated and fed back into step 1 recursively until a single result remains.\n3. **Graceful truncation**: Single items exceeding the character limit are truncated with an ellipsis marker rather than failing. In pathological cases where every item individually exceeds the per-item budget, all items are uniformly truncated so processing can continue.\n4. **Context cancellation support**: The algorithm checks for context errors at entry points and before each recursive reduction round, allowing clean shutdown during long-running synthesis.\n\n### Note on `chunkTextWithOverlap`\n\nThis function is a text-splitting utility with overlap between adjacent chunks. It does not participate in the main reduction pipeline shown here; it appears to be available for reuse elsewhere in the codebase but is unused in this file.\n\n#### [STATE_AND_CONCURRENCY]\n**Mutable State:** No mutable state (no package-level/global variables). All function parameters (`items`, `cfg`) and local variables (`batches`, `currentBatch`, etc.) are stack-allocated or passed by value; mutations to `items[0]` occur within a single call chain and do not escape.\n\n**Concurrency Mechanisms:** None. No `sync.Mutex`, channels, atomic operations, or other concurrency primitives appear in this file.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects \u0026 Error Handling Analysis β€” `chunking.go`\n\n---\n\n### 1. External Communication (I/O)\n\nThis file has **no direct external I/O**. It does not open files, make network calls, or talk to a database directly. All external communication is delegated through the `llmCaller` interface:\n\n| Function | External Action | Mechanism |\n|---|---|---|\n| `reduceWithLLM` | LLM API call | Delegates to `c.CallLLM(ctx, ...)` β€” an injected dependency (`llmCaller`). No direct network I/O in this file. |\n| `reduceInChunks`, `reduceFileFacts` | Indirectly invoke LLM calls | Both ultimately call `reduceWithLLM`, which invokes `c.CallLLM`. The actual network I/O happens inside the caller implementation, not here. |\n\n**Logging is internal only.** The `logMsg` callbacks passed in produce log messages (e.g., `\"➜ LLM Synthesizing chunk for %s (%d items)\"`), but these are just formatted strings returned to a callback β€” no actual log writer is called from this file. No disk, DB, or network writes occur here.\n\n---\n\n### 2. Error Handling \u0026 Returns\n\n#### 2.1. Propagation via `error` return value\n\nEvery function returns `(string, error)`. Errors propagate up through the call chain without being caught locally in most cases:\n\n- **`reduceWithLLM`** β€” Wraps LLM errors with context using `fmt.Errorf(\"%s: %w\", errMsg, err)` where `errMsg` is a caller-provided string (e.g., `\"LLM error during synthesis\"`). This is a **wrapper pattern**, not a panic.\n- **`reduceInChunks`, `reduceFileFacts`** β€” Errors from `reduceWithLLM` propagate directly to the caller unchanged. No wrapping added by these functions themselves.\n- **`reduceItems`** β€” Propagates errors from recursive calls via bare `return \"\", err`. If a batch fails, the entire reduction short-circuits and returns the error.\n\n#### 2.2. Context cancellation\n\n```go\nif err := ctx.Err(); err != nil {\n return \"\", err\n}\n```\n\nChecked at two points in `reduceItems`: before batching and inside the recursive loop. Cancellation errors propagate straight up β€” no wrapping, just pass-through.\n\n#### 2.3. Silent data loss / truncation (no error)\n\nSeveral paths silently truncate or drop input without returning an error:\n\n| Path | Behavior |\n|---|---|\n| Empty items (`len(items) == 0`) in `reduceWithLLM` | Returns `\"\", nil` β€” caller gets success with empty string. |\n| Single item exceeding `maxChars` | Truncated to `maxChars + \"\\n...[truncated]\"`, **no error**. |\n| All items individually exceed `allowedPerItem` (infinite-loop guard) | Each item truncated in place, batched as one group, then reduced β€” **no error**. |\n| `chunkTextWithOverlap`: `maxRunes \u003c= 0` or `step \u003c= 0` | Returns the original text unchanged (`return []string{text}`), no error. |\n\nThese truncations are **data-loss side effects** that produce a partial result but signal success. The caller has no way to distinguish \"successfully reduced\" from \"partially truncated and reduced.\"\n\n#### 2.4. No panics\n\nNo `panic` calls in this file. All failure modes use the standard `(string, error)` return convention or silent data truncation.\n\n---\n\n### Summary Table\n\n| Concern | How It's Handled |\n|---|---|\n| Network / external I/O | Delegated to `llmCaller.CallLLM`; none in this file. |\n| File / DB writes | None. |\n| Logging | Internal string formatting; no actual log sink called here. |\n| Error wrapping | LLM errors wrapped with caller-supplied prefix via `%w`. Other errors propagated raw. |\n| Context cancellation | Returned as-is, unwrapped. |\n| Silent truncation | Multiple paths truncate data without error β€” a notable side effect with no signal to the caller. |\n| Panics | None. |" + }, + "internal/engine/client.go": { + "sha256": "998aa6b162f007326e91cb271e6fc0cc59413378394cd887f85b8a4a832487cb", + "facts": "#### [API_SIGNATURES]\n## Public Surface Area of client.go\n\n### Structs\n\n- **`Message`**\n - Fields: `Role string`, `Content string`\n\n#### [BUSINESS_LOGIC]\n## Domain Concepts\n\n**LLM Inference Client.** This file implements a structured interface for communicating with a Local Large Language Model (Ollama) via its REST API. The business rule is: given a system prompt and user messages, send them to an Ollama chat endpoint and return the model's text response β€” or report failure if communication breaks down.\n\n## High-Level Algorithmic Flow\n\n1. **Prepare payload.** Assemble the request body by prepending the `systemPrompt` as a system-role message before any user-supplied messages, then serialize it as JSON to the Ollama `/api/chat` endpoint.\n2. **Send HTTP request.** Issue a single synchronous POST request with a fixed timeout; no retries are performed β€” failures propagate immediately (fail-fast policy).\n3. **Handle response.** On `200 OK`, deserialize the body and extract the model's reply from the `message.content` field. Any non-2xx status code is read into an error string containing both the HTTP status and raw response payload.\n4. **Return result.** Either the extracted text content, or a wrapped error describing what went wrong.\n\n## Business Rules\n\n- **Single-shot calls only.** The caller receives no retry logic; if Ollama is unreachable or times out, the operation returns an error immediately.\n- **Structured message ordering.** System prompt must be injected first as a system-role entry before user messages, preserving chat context semantics expected by Ollama.\n- **Error reporting preserves diagnostic data.** Non-success responses are not silently swallowed β€” their status code and body are returned to aid debugging.\n\n#### [STATE_AND_CONCURRENCY]\n**Mutable State:** None. All struct fields (`modelID`, `baseURL`, `numCtx`, `httpClient`) in `llmClient` are assigned once during construction and never modified afterward. The methods only read from these fields or operate on local variables.\n\n**Concurrency Mechanisms:** None. No mutexes, channels, atomics, or other synchronization primitives appear in this file.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects \u0026 Error Handling Analysis\n\n---\n\n### External Communication (Side Effects)\n\n| Operation | Location | Type |\n|-----------|----------|------|\n| `http.NewRequestWithContext` / `c.httpClient.Do(req)` | `prepareOllamaRequest`, `CallLLM` | **Network I/O** β€” HTTP POST to LLM provider endpoint (`\u003cbaseURL\u003e/api/chat`) |\n\n- No disk, file system, or database access.\n- In-memory only: `bytes.NewReader(bodyBytes)` is a memory-backed reader (not actual file I/O).\n- The `*http.Client` carries a configured timeout (`defaultHTTPTimeout`, value not defined in this file).\n\n---\n\n### Error Handling Patterns\n\n#### 1. Marshaling Failure β€” wrapped with `%w`\n\n```go\nbodyBytes, err := json.Marshal(reqBody)\nif err != nil {\n return nil, fmt.Errorf(\"failed to marshal request: %w\", err)\n}\n```\n\nWraps the underlying error so callers can inspect it via `errors.Is`.\n\n#### 2. HTTP Request Construction Failure β€” wrapped with `%w`\n\n```go\nreq, err := http.NewRequestWithContext(ctx, \"POST\", url, bytes.NewReader(bodyBytes))\nif err != nil {\n return nil, err\n}\n```\n\nThis one is **not** wrapped β€” it returns the raw error from `http.NewRequestWithContext`. Unwrapped errors make unwrapping impossible for callers.\n\n#### 3. Network/Transport Errors β€” returned as-is (unwrapped)\n\n```go\nresp, err := c.httpClient.Do(req)\nif err != nil {\n return \"\", err\n}\n```\n\nNo wrapping. Callers cannot distinguish between timeout, DNS failure, connection reset, etc., via `errors.Is`.\n\n#### 4. Response Body Read on Success Path β€” returned as-is (unwrapped)\n\n```go\nrespData, err := io.ReadAll(resp.Body)\nif err != nil {\n return \"\", err\n}\n```\n\nUnwrapped. Likely rare in practice (network errors already caught above), but still unwrapped.\n\n#### 5. JSON Unmarshaling Failure β€” wrapped with `%w`\n\n```go\nvar result ollamaResponse\nif err := json.Unmarshal(respData, \u0026result); err != nil {\n return \"\", fmt.Errorf(\"failed to parse response: %w\", err)\n}\n```\n\nWrapped. Callers can inspect the underlying error.\n\n#### 6. Non-OK Status Codes β€” wrapped with custom message (not `%w`)\n\n```go\nbody, _ := io.ReadAll(io.LimitReader(resp.Body, maxErrorBodyBytes))\nreturn \"\", fmt.Errorf(\"ollama api error: status %d, response: %s\", resp.StatusCode, string(body))\n```\n\n- The `_` discards the read error β€” **information lost**.\n- Wrapped with a custom `fmt.Errorf(...)` message (not `%w`). Callers cannot unwrap to find the original transport-level error.\n- Uses `maxErrorBodyBytes` which is **undefined** in this file β€” will fail at compile time if not imported from somewhere else.\n\n---\n\n### Summary Table\n\n| Path | Wraps Error? | Provides Context? | Notes |\n|------|-------------|-------------------|-------|\n| Marshal failure | βœ… `%w` | Yes | Good |\n| HTTP request construction | ❌ β€” raw | No | Unwrapped, context lost |\n| Network/transport error | ❌ β€” raw | No | Unwrapped; caller can't distinguish timeout vs DNS vs connect |\n| Success-path body read | ❌ β€” raw | No | Unwrapped (rare to trigger) |\n| JSON unmarshal failure | βœ… `%w` | Yes | Good |\n| Non-OK status response | ❌ custom message | Partial (status code only) | Discards read error; uses undefined constant |\n\n---\n\n### Additional Observations\n\n- **No retry logic** β€” errors are returned immediately (\"failing fast without retries\" per comment). A network blip becomes a full failure.\n- **`defaultHTTPTimeout` and `maxErrorBodyBytes`** are referenced but not defined in this file. If they come from another package, callers have no visibility into their values or how they're set.\n- **No context propagation check for cancellation between Do() and Body.Close()** β€” a cancelled context during the HTTP call will still trigger `defer resp.Body.Close()` which may return an error on close (silently ignored via `defer` with no recovery)." + }, + "internal/engine/constants.go": { + "sha256": "85ec25e43f84ebf8a1ce483e8efd9df8eb8fcf20fa2e8bbf3e243022a0730ae7", + "facts": "#### [API_SIGNATURES]\n## Public Surface Area: `internal/engine/constants.go`\n\n### Exported Types\n\n| Name | Kind | Signature / Description |\n|---|---|---|\n| [`LogEventFunc`](#logeventfunc) | Type Alias | β€” |\n\n#### LogEventFunc\n\n```go\ntype LogEventFunc func(EventType, string)\n```\n\n- **Input**: `EventType` (parameter), `string` (parameter) \n *(exact argument types are unexported and not declared in this file)*\n- **Output**: `void` β€” function signature only\n\n#### [BUSINESS_LOGIC]\n## Business Logic \u0026 Domain Concepts\n\n### What this file solves\n\nThis file has **no business logic**. It defines configuration parameters and type signatures consumed by other files in the `internal/engine` module.\n\n---\n\n### Constants Defined\n\n| Constant | Value | Purpose (inferred from name) |\n|---|---|---|\n| `defaultHTTPTimeout` | 10 minutes | Default timeout for HTTP client requests |\n| `maxErrorBodyBytes` | 1024 bytes | Maximum size allowed in error response bodies |\n| `defaultChunkOverlap` | 800 chars | Overlap between consecutive chunks during streaming/processing |\n| `minNumCtxFloor` | 512 tokens | Minimum context window floor for processing |\n| `contextWindowAllocRatio` | 0.75 | Ratio used when allocating context windows (likely reserves 75% of available space) |\n| `maxCharsMultiplier` | 3 | Multiplier applied to character counts (likely scales token estimates or max output limits) |\n| `metadataFileName` | `.metadata.json` | Persistent metadata file name for the engine's state |\n| `agentsFileName` | `AGENTS.md` | Agent definitions/configuration file |\n| `defaultDirPerm` | 0755 | Default directory permission mode |\n\n### Type Defined\n\n- **`LogEventFunc`** β€” A function type with signature `func(EventType, string)`. This is a callback interface for logging events (no business rules here).\n\n---\n\n### High-Level Algorithmic Flow\n\n**None.** This file contains no algorithms. It provides named constants that other engine components reference when making decisions about timeouts, chunking behavior, resource allocation ratios, and filesystem conventions.\n\n#### [STATE_AND_CONCURRENCY]\n**Mutable State:** None. The file defines only constants and a function type alias (`LogEventFunc`). No global variables, struct fields, or any runtime-mutable values exist in this scope.\n\n**Concurrency Mechanisms:** None. No `sync.Mutex`, channels, atomic operations, or other concurrency primitives are used.\n\n---\n*Note: Constants like `defaultHTTPTimeout` and `maxErrorBodyBytes` are compile-time constants with no runtime mutation.*\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects \u0026 Error Handling Analysis\n\n### I/O Communication\n**None detected.** This file contains zero external communication:\n- No network calls, database queries, or API requests\n- No disk read/write operations (`.metadata.json`, `AGENTS.md` are only stored as constant strings)\n- No external resource access β€” these filenames exist solely to be referenced by other code in the package\n\n### Error Handling\n**None detected.** This file contains zero error-handling logic:\n- No panic recovery or deferred calls\n- No error wrapping, sentinel values, or custom error types\n- `maxErrorBodyBytes = 1024` is a size limit constant β€” it does not implement any actual error-body truncation or validation\n\n### Summary\nThis file is purely declarative. It defines:\n- **Constants** for timeouts, sizes, ratios, and filenames (`time`, bytes, percentages, permission modes)\n- **One type alias**: `LogEventFunc` β€” a function signature `(EventType, string)` that *could* be used for logging side effects elsewhere in the package, but this file does not instantiate or call any such handler\n\nAll behavior is deferred to other files. No runtime state changes occur here." + }, + "internal/engine/json_parser.go": { + "sha256": "81bea4827161eb8978cd6f288e2276c2616da7832e8687400b6872d8d9c3749c", + "facts": "#### [API_SIGNATURES]\n## Public Surface Area of `json_parser.go` (Module: `internal/engine`)\n\n| Exported Item | Type | Input / Output |\n|---|---|---|\n| `stripOuterMarkdownFence` | Function | `string` β†’ `string` |\n\n#### [BUSINESS_LOGIC]\n## File: `json_parser.go` (Module: `internal/engine`)\n\n### Domain Concept\nMarkdown/JSON code fence extraction β€” removing surrounding triple-backtick fences from input strings to retrieve the raw inner content.\n\n### Business Rule / Algorithmic Flow\n1. **Trim whitespace** from the input string.\n2. **Attempt regex match** against a pattern that captures:\n - Triple backticks (` ``` `) with 3+ characters, optionally followed by `markdown` or `json`.\n - All content between the opening and closing fences (captured in group 1).\n3. **If matched**: return the trimmed captured inner content.\n4. **If no match**: return the original trimmed input unchanged.\n\n### Summary\nA single-pass parser that strips markdown or JSON code fences from strings, returning either the extracted payload or the untouched input when no fence is present.\n\n#### [STATE_AND_CONCURRENCY]\n**Mutable State:** None. No global variables are mutated after initialization (`markdownFenceRe` is read-only). The function does not modify any package-level or struct fields.\n\n**Concurrency Mechanisms:** None.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects (I/O Communication)\n\n**None.** This file contains zero external communication:\n\n- No network calls, disk reads/writes, database queries, or any `os`/`io` package usage.\n- The only \"external\" dependency is the standard library packages `regexp` and `strings`, which are internal to Go's runtimeβ€”no outside-world I/O occurs during execution.\n- `markdownFenceRe` is compiled once at package init via `regexp.MustCompile`. This is a one-time in-memory state initialization, not an I/O operation.\n\n## Error Handling\n\n**None.** The function communicates errors implicitly by returning no match:\n\n- **No error return parameter.** `stripOuterMarkdownFence` returns only the extracted string (or the original trimmed input if no fence matches).\n- **No panic recovery or wrapping.** If a panic occurs inside the regex engine, it is unwrapped and propagates to the callerβ€”there is no catch.\n- **No sentinel values.** There is no custom error type or sentinel constant defined in this file.\n- **Silent fallback.** On non-match, `strings.TrimSpace(matches[1])` simply returns the original trimmed input unchangedβ€”the \"error\" of not finding a fence is treated as a no-op and swallowed.\n\n**One caveat:** `regexp.MustCompile` panics if the pattern itself is invalid at package init time. That panic is unhandled within this file and would propagate to any caller that imports the package." + }, + "internal/engine/orchestrator.go": { + "sha256": "0a747df1552aa7a6fb8f024c5ef1cc52284551ef0aeef08d9e4d9bb83b08c7a8", + "facts": "#### [API_SIGNATURES]\n# Public Surface Area: `internal/engine/orchestrator.go`\n\n## Exported Types\n\n### `EventType` (type)\n```go\ntype EventType string\n```\n\n### Constants on `EventType`\n\n| Constant | Value |\n|---|---|\n| `EventStatus` | `\"status\"` |\n| `EventError` | `\"error\"` |\n\n### `Event` (struct)\n```go\ntype Event struct {\n Type EventType\n Message string\n}\n```\n\n## Unexported Types \u0026 Methods (Internal API β€” Not Public)\n\nThe following type and methods are **not** part of the public surface because their receiver/type is unexported (`orchestrator`):\n\n- `orchestrator` struct\n- `(o *orchestrator).GenerateStandardDocs(...)` \n- `(o *orchestrator).RunInit(...)`\n- `(o *orchestrator).RunUpdate(...)`\n- `setupPipeline(...)` β€” standalone helper (unexported)\n- `teardownPipeline(...)` β€” standalone helper (unexported)\n\n## Standalone Functions Referenced but Not Defined Here\n\nThese types are used in function signatures but defined elsewhere:\n\n| Type | Package | Notes |\n|---|---|---|\n| `LogEventFunc` | `engine` | Function type passed as parameter to exported methods; signature not shown |\n| `pipelineContext` | `engine` | Internal context struct (unexported) |\n| `DirNode`, `FileCacheEntry`, `FileChange` | external packages (`internal/tools`, etc.) | Referenced but definitions outside this file |\n\n#### [BUSINESS_LOGIC]\n## Domain Concepts \u0026 Business Rules\n\n### Core Entity: Metadata Cache (`MetadataCache`)\nA persistent state object that survives across pipeline invocations. It stores:\n- **`Files`** β€” map of file path β†’ `FileCacheEntry` (cached hash + last synthesis step)\n- **`Modules`** β€” map of module directory paths β†’ their synthesized summaries\n- **`StepsHash`** β€” derived from the configured extraction steps; acts as a version key\n\n### Core Concept: Extraction Steps Hashing Rule\nThe cache is invalidated *whenever* `StepsHash != currentStepsHash`. This is an all-or-nothing rule: any change to the extraction pipeline configuration forces full cache invalidation (resetting both `Files` and `Modules`). This prevents stale synthesis results from persisting after a config change.\n\n### Core Concept: File Change Detection via SHA256\nEach code file's current state is verified by computing its SHA256 hash at runtime. The cache stores the last-known hash per file. Classification into three states:\n- **Added** β€” in current discovery, not in cache\n- **Modified** β€” present in both but hashes differ\n- **Deleted** β€” in cache but absent from current discovery (file removed from disk)\n\n### Core Concept: Tree-Based Affected-Region Propagation\nAfter computing changed files, the system builds a directory tree and marks directories containing changes. This is then *propagated* upward through parent directories. The business rule here is: **any change to a file implies all ancestor directories need re-synthesis**.\n\n### Core Concept: Stale Module Cleanup Rule\nModule summaries stored in `cache.Modules` are validated against the current active directory tree. Any module path not represented in the live tree has its cached summary removed *and* its physical `.md` file deleted from disk (if it exists). This is a purge rule β€” stale documentation is actively cleaned up, not left behind.\n\n### Core Concept: Standard Docs Generation Rule\nThe `architecture.md` and `quickstart.md` are generated in full when either:\n- The root directory (`.`) was affected by changes, OR\n- Neither file exists on disk yet (first-run scenario), OR\n- Both exist but the user explicitly triggers regeneration\n\nWhen neither condition holds, the existing architecture/quickstart pages are preserved β€” they are treated as globally stable and not subject to incremental updates.\n\n## High-Level Algorithmic Flow\n\n### `RunInit` (Cold Start)\n```\n1. Setup pipeline β†’ discover code files, compute hashes, build tree\n2. Mark every directory as affected (full regen scenario)\n3. Synthesize root summary from the full tree via LLM calls\n4. Generate architecture.md + quickstart.md using root summary\n5. Write/update agent guidelines file (appending if not already present)\n6. Persist cache to disk β†’ complete\n```\n\n### `RunUpdate` (Incremental Refresh)\n```\n1. Setup pipeline β†’ discover current code files, compute hashes\n2. Invalidate entire cache if extraction steps changed\n3. Classify all files into Added/Modified/Deleted buckets by comparing current vs cached hashes\n4. Build directory tree from allowed (existing + new) files only\n5. Purge stale module summaries and their physical files for any path no longer in the active tree\n6. Compute affected directories β†’ propagate upward through parent dirs\n7. If nothing changed: return early (documentation is up to date)\n8. Otherwise: synthesize root summary from the affected region\n9. Re-generate architecture.md + quickstart.md only if affected or missing\n10. Persist cache β†’ complete\n```\n\n### Shared Pipeline Context (`pipelineContext`)\nBoth entry points construct a context bundle containing: current LLM client, repository root, config, cache, affected directory set, precomputed file hashes, and the event logger β€” then pass it to `synthesizeNode` for tree traversal and synthesis.\n\n#### [STATE_AND_CONCURRENCY]\n## Mutable State Analysis: `orchestrator.go` (internal/engine)\n\n### Mutable Struct Fields\n\n1. **`*MetadataCache`** β€” mutated across multiple fields and methods:\n - `.StepsHash` β€” set in `setupPipeline`, updated with new hash in `RunUpdate` when extraction steps change\n - `.Files` map β€” cleared and rebuilt in `RunUpdate` (new map literal assigned)\n - `.Modules` map β€” entries deleted in `RunUpdate` when active directories don't match\n\n2. **`pipelineContext`** fields β€” created fresh per call but contain mutable state:\n - `.affectedDirs`, `.precalculatedHashes`, `.cache`, etc. β€” all initialized new each invocation\n - No shared instance; context is passed by value or pointer consistently, not reused across goroutines in this file\n\n3. **`orchestrator`** struct fields β€” `client llmCaller` field appears to be set once at construction (no mutation observed in this file)\n\n### Global Variables / Package-Level State\n- **None visible** in this file\n\n### Concurrency Protection Mechanisms\n- **No mutexes, channels, atomic operations, or other concurrency primitives** used anywhere in this file\n- No `sync.Mutex`, `sync.RWMutex`, `atomic.Value`/`atomic.Bool`/`atomic.Int32` etc. present\n- No channel-based synchronization\n\n### Concurrency Risk Assessment\n- **No explicit protection** for the mutable state identified above\n- The code is single-threaded in appearance (no goroutine launches or channel usage)\n- If `pipelineContext` were shared across goroutines, there would be a data race due to unprotected writes to `.cache`, `.affectedDirs`, etc. β€” but no such sharing occurs within this file's scope\n\n### Summary\n| Category | Status |\n|---|---|\n| Global variables | None (in this file) |\n| Struct fields with mutation | `MetadataCache` (`.StepsHash`, `.Files`, `.Modules`), `pipelineContext` (local instances only) |\n| Concurrency primitives | **None** β€” no mutex, channel, or atomic protection present |\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n# Side Effects \u0026 Error Handling Analysis β€” `orchestrator.go`\n\n## External Communication (I/O)\n\n### Network / LLM Calls\n| Location | What Happens | Context |\n|---|---|---|\n| `o.client.CallLLM(...)` in `GenerateStandardDocs` | Two calls: one for architecture.md, one for quickstart.md. Each call is made with the system prompt + a user message describing what to generate. | Network-bound; depends on an external model provider. Errors are wrapped and returned. |\n| Implicit LLM calls via `synthesizeNode` (reached from `RunInit` / `RunUpdate`) | Recursive tree synthesis eventually delegates to `o.client.CallLLM`. | Same network dependency. Errors propagate up through the chain. |\n\n### Disk I/O β€” Write Operations\n| Location | What Happens | Notes |\n|---|---|---|\n| `tools.WriteFileSafely(repoRoot, archPath, ...)` | Writes architecture.md to disk. Markdown fence is stripped before writing. | Error wrapped and returned. |\n| `tools.WriteFileSafely(repoRoot, qsPath, ...)` | Writes quickstart.md to disk. Same strip logic applied. | Error wrapped and returned. |\n| `tools.WriteFileSafely` in `RunInit` (agent guidelines) | Either writes a fresh agent guidelines file or appends it if the expected marker is missing. | Logic distinguishes \"file doesn't exist\" vs \"exists but wrong content\". Write error is wrapped. |\n\n### Disk I/O β€” Read Operations\n| Location | What Happens | Notes |\n|---|---|---|\n| `tools.LoadGitignore(repoRoot)` in `setupPipeline` | Reads `.gitignore` from repo root to collect ignore patterns. | **Error swallowed**: logged as warning, no return error. Pipeline continues with empty pattern list if this fails. |\n| `tools.DiscoverCodeFiles(repoRoot, ignores)` in both entry points | Walks filesystem to discover code files matching configured filters. | Errors are propagated up (not swallowed). |\n| `computeSHA256(repoRoot, f)` per file | Reads each discovered file's content and computes its SHA-256 hash. | **Error swallowed**: logged as warning for that specific file; the file is excluded from downstream processing but pipeline continues. |\n| `tools.ReadFileSafely` in `RunInit` (agent guidelines) | Reads existing agent file to decide whether to overwrite or append. | Error wrapped and returned if write fallback also fails. No explicit recovery path shown β€” it just writes a new file on failure, which may be inconsistent with the read outcome. |\n| `security.SafeResolve` + `os.Stat` in `RunUpdate` (arch/quickstart checks) | Resolves paths to architecture.md / quickstart.md and checks existence before deciding whether to regenerate standard docs. | Errors from SafeResolve are checked; if resolution fails, `archExists`/`qsExists` default to false β†’ triggers regeneration. This is a **silent assumption**: it treats \"can't resolve\" as \"file does not exist\". |\n| `tools.WriteFileSafely(repoRoot, agentsFileName, ...)` in teardown? / RunInit fallback | Writes agent guidelines on failure of ReadFileSafely. | Effectively replaces whatever was read with a fresh file β€” **data loss** for any existing content that isn't the expected marker. |\n| `os.Remove(absModuleFile)` (via `security.SafeResolve`) in `RunUpdate` | Removes stale module markdown files when cache is invalidated and directory no longer active. | **Error swallowed**: `_ = os.Remove(...)` discards the error entirely. Stale files may be left behind if removal fails. |\n\n### Disk I/O β€” Directory Operations\n| Location | What Happens | Notes |\n|---|---|---|\n| `os.MkdirAll(modulesDir, defaultDirPerm)` in `RunInit` | Creates the `modules/` subdirectory inside docsDir. | Error wrapped and returned. |\n\n### Cache I/O (via external functions)\n- `loadMetadataCache(repoRoot, docsDir)` β€” reads a metadata cache file from disk. Failure swallowed and logged as warning.\n- `saveMetadataCache(repoRoot, docsDir, cache)` β€” writes the cache back to disk in teardown. Failure swallowed and logged as warning.\n\n---\n\n## Error Handling Patterns\n\n### 1. Wrapped Errors (`fmt.Errorf(\"...: %w\", err)`)\nUsed for **critical failures** where recovery is not possible or appropriate:\n- LLM call failure β†’ wrapped with context (\"failed to generate global architecture\", \"failed to generate quickstart documentation\")\n- File write failure β†’ wrapped (\"failed to write architecture.md\", etc.)\n- Directory creation failure β†’ wrapped (\"failed to create modules directory\")\n- Agent guidelines write failure (both fresh-write and append paths)\n\n### 2. Silently Swallowed Errors + Warning Log\nUsed for **non-critical, recoverable** failures where the pipeline should continue:\n- `.gitignore` load failure\n- Metadata cache load failure\n- Per-file SHA-256 computation failure (file skipped from hash map, excluded from change detection)\n\nPattern: `logEvent(EventStatus, fmt.Sprintf(\"Warning: ...\", err))` β€” no return error.\n\n### 3. Error Propagation Up the Chain\nFunctions that cannot meaningfully continue after a hard failure propagate errors:\n- `GenerateStandardDocs`: returns wrapped error on LLM or write failure\n- `RunInit`, `RunUpdate`: both return errors from their main pipeline steps (file discovery, tree synthesis)\n\n### 4. Short-Circuit / No-Op on Success\n- `RunUpdate` checks if any files changed (`len(affectedDirs) == 0`). If no changes, returns `nil` immediately without running the full pipeline. This is an **optimization**, not error handling β€” but it means errors from earlier steps (like cache invalidation logic) are also skipped on \"no change\" paths.\n\n### 5. Conditional Regeneration\nIn `RunUpdate`, standard docs regeneration is conditional:\n```go\nif affectedDirs[\".\"] || !archExists || !qsExists {\n // regenerate\n} else {\n logEvent(EventStatus, \"Global architecture and quickstart pages are up to date. Skipping regeneration.\")\n}\n```\nThis means if neither root nor any other directory was affected AND the existing files can be resolved β†’ standard docs are not regenerated. If either check fails (e.g., file missing or resolution error), it regenerates.\n\n### 6. No Panic Statements Found\nNo `panic` calls exist in this file. All errors go through explicit return values or logging + swallow patterns.\n\n---\n\n## Summary Table\n\n| Category | Pattern | Count / Frequency | Risk Level |\n|---|---|---|---|\n| Network (LLM) | Wrapped error on failure β†’ propagated up | 2 direct + recursive | High β€” pipeline stops if model is unreachable or prompt fails |\n| Disk Write | Wrapped error on failure β†’ propagated up | Multiple explicit calls | Medium β€” docs not written, caller sees failure |\n| Disk Read (.gitignore) | Swallowed β†’ warning log | 1 call in `setupPipeline` | Low β€” just loses ignore patterns |\n| Disk Read (SHA256 per file) | Swallowed β†’ warning log + skip file | Per-file loop | Medium β€” changed files may be missed if hash computation fails silently |\n| Disk Write (Agent Guidelines) | Fallback write on read failure; append logic | 1 call in `RunInit` | **High** β€” existing content replaced if marker missing, even if original was valid |\n| Disk Remove (Module Files) | Swallowed (`_ = os.Remove(...)`) | 1 call in `RunUpdate` | Medium β€” stale files persist if removal fails |\n| Cache I/O | Swallowed + warning log | load/save cache | Low β€” cache inconsistency possible if save fails |\n| Path Resolution | Silent assumption (treat failure as \"not exists\") | 2 calls in `RunUpdate` | **High** β€” could trigger unnecessary regeneration or skip existence checks incorrectly |\n\n---\n\n## Notable Risks / Observations\n\n1. **Agent guidelines write-on-read-failure**: If reading the agent file fails, it writes a fresh file. This is functionally fine if you consider \"can't read = needs to be written\", but it silently replaces any existing content that happened to not contain the marker β€” which could lose user-added content.\n\n2. **Cache save failure in teardown**: `saveMetadataCache` error is swallowed. If the pipeline succeeds but cache isn't saved, subsequent runs will rebuild from scratch instead of using cached state. This is a silent data loss for future invocations.\n\n3. **Module file removal silently fails**: Stale module files may not get cleaned up if `os.Remove` returns an error (e.g., permission denied on Windows). No retry or alternative cleanup path exists.\n\n4. **No transactional safety around LLM + write**: The architecture and quickstart docs are generated sequentially via two separate LLM calls followed by two writes. If the second LLM call succeeds but the first write failed, you end up with an inconsistent state: architecture.md missing but quickstart.md present (or vice versa)." + }, + "internal/engine/runner.go": { + "sha256": "e833cb64ae16b96a2b0ba3ce4fb6d4f9010c49809a5685ba283b6a05b34464ee", + "facts": "#### [API_SIGNATURES]\n# Public Surface Area: `runner.go` (package `engine`)\n\n## Types\n\n### `Mode`\n\n```go\ntype Mode string\n```\n\n**Kind:** Type alias (`string`).\n\n---\n\n### `Runner`\n\n```go\ntype Runner struct {\n cfg *config.Config\n}\n```\n\n**Kind:** Struct.\n\n| Field | Type | Nullable |\n|-------|------|----------|\n| `cfg` | `*config.Config` | Yes |\n\n---\n\n## Constants\n\n### `ModeInit`\n\n```go\nconst ModeInit = \"init\"\n```\n\n**Value:** `\"init\"`\n\n---\n\n### `ModeUpdate`\n\n```go\nconst ModeUpdate = \"update\"\n```\n\n**Value:** `\"update\"`\n\n---\n\n## Functions\n\n### `NewRunner`\n\n```go\nfunc NewRunner(cfg *config.Config) *Runner\n```\n\n| Param | Type | Nullable |\n|-------|------|----------|\n| `cfg` | `*config.Config` | Yes |\n\n**Returns:** `*Runner`\n\n---\n\n## Methods\n\n### `(Runner).Run`\n\n```go\nfunc (r *Runner) Run(ctx context.Context, repoRoot string, mode Mode, onEvent func(Event)) error\n```\n\n| Param | Type | Nullable |\n|-------|------|----------|\n| `ctx` | `context.Context` | No |\n| `repoRoot` | `string` | No |\n| `mode` | `Mode` | No |\n| `onEvent` | `func(Event)` | No |\n\n**Returns:** `error`\n\n#### [BUSINESS_LOGIC]\n## Business Rules \u0026 Domain Concepts\n\n**Domain Concept β€” Documentation Pipeline Modes:** The system supports two distinct modes of operation: `init` (full documentation generation) and `update` (incremental updates). Mode selection is a branching decision at runtime.\n\n**Business Rule β€” Repository Locking:** Before any pipeline work begins, the runner must acquire an exclusive lock on the repository root. If locking fails, it returns an error immediatelyβ€”no partial execution proceeds. This prevents concurrent documentation runs from interfering with each other.\n\n**Business Rule β€” Gitignore Maintenance:** As a pre-flight check, the runner attempts to ensure the lockfile is listed in `.gitignore`. Any failure here is logged as a warning but does not block execution. The pipeline continues regardless.\n\n## High-Level Algorithmic Flow\n\n1. **Lock acquisition** β†’ attempt exclusive lock on repo root; fail fast if unobtainable.\n2. **LLM client + orchestrator construction** β†’ instantiate the LLM client (using configured model ID, base URL, context count) and wire it to an orchestrator.\n3. **Mode dispatch** β†’ if `init`, call `orch.RunInit`; if `update`, call `orch.RunUpdate`. Any other mode returns a \"unsupported\" error.\n4. **Pipeline execution** β†’ the selected orchestration method runs its full flow.\n5. **Lock release** β†’ deferred unlock occurs after completion (success or failure).\n\n#### [STATE_AND_CONCURRENCY]\n## Mutable State Analysis\n\n### Mutable State Found:\n\n| Location | Field/Variable | Type | Mutability |\n|----------|---------------|------|------------|\n| `Runner` struct | `cfg` | `*config.Config` (pointer) | Potentially mutable β€” pointer field can be reassigned via dereferencing, though no such mutation occurs in this file. Set once in `NewRunner`. |\n\n### Concurrency Mechanisms:\n\n**None.** No `sync.Mutex`, channels, atomic operations, or other concurrency primitives are used in this file. The deferred `lock.Unlock()` call suggests an external lock acquired via `security.AcquireLock()`, but its implementation and protection scope are not visible here; it cannot be confirmed to protect any shared mutable state within this module.\n\n### Summary:\n- **1 mutable field** (`cfg` pointer on `Runner`) β€” no mutation observed in this file.\n- **No concurrency primitives** declared or used explicitly.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects \u0026 External Communication\n\n### Filesystem I/O (Disk)\n\n| Location | Action | Fatal? |\n|---|---|---|\n| `security.EnsureGitignoreHasLockfile(repoRoot)` | Reads the `.gitignore` file at `repoRoot` and appends a lockfile entry | **No** β€” error is swallowed; only logged via the `onEvent` callback as `\"Warning: failed to ensure gitignore has lockfile: %v\"` |\n| `security.AcquireLock(repoRoot)` | Reads/writes lock files under `repoRoot` (likely `.git.lock` or similar) | **Yes** β€” error is wrapped and returned to caller as `\"failed to acquire repository lock: \u003cerr\u003e\"` |\n| `defer lock.Unlock()` | Releases the acquired lock file when `Run` returns | Implicit cleanup; no separate error path |\n\n### Network I/O (LLM Client)\n\n- `newLLMClient(r.cfg.ModelID, r.cfg.OllamaBaseURL, r.cfg.OllamaNumCtx)` constructs an LLM client that connects to `OllamaBaseURL`. All subsequent calls through this client (`orch.RunInit`, `orch.RunUpdate`) will perform HTTP/network I/O. Errors from these network interactions propagate up through the orchestrator and into `Run` as returned errors (unwrapped).\n\n### Event Callback (Process-level)\n\n- `onEvent func(Event)` is passed in by the caller. The runner uses it to emit a warning for the gitignore failure, but otherwise does not invoke it further within this method.\n\n---\n\n## Error Handling / Return Paths\n\nThe function returns errors from **four** distinct paths:\n\n1. **Lock acquisition failure** β€” wrapped with `fmt.Errorf(\"failed to acquire repository lock: %w\", err)`. Caller receives a new error wrapping the original.\n2. **Init mode failure** β€” returned as-is from `orch.RunInit(ctx, repoRoot, r.cfg, onEvent)`. No additional wrapping.\n3. **Update mode failure** β€” returned as-is from `orch.RunUpdate(ctx, repoRoot, r.cfg, onEvent)`. No additional wrapping.\n4. **Unsupported mode** β€” returns a plain string error: `\"unsupported mode: %s\"`.\n\n### Swallowed / Non-Fatal Errors\n\n- The gitignore warning is the only case where an external I/O failure is not returned to the caller. It is logged via `onEvent(EventStatus, ...)` and execution continues into step 2 (lock acquisition). If the lock fails after this point, it is treated as fatal." + }, + "internal/engine/synthesize.go": { + "sha256": "5940958793755d9fc83d3d9643536f9a4ef121346147f41cdbd1294f2b771ff7", + "facts": "#### [API_SIGNATURES]\n# Public Surface Area: Module `internal/engine`, File `synthesize.go`\n\n**No exported items found.** All types and functions in this file use lowercase identifiers (`pipelineContext`, `synthesizeNode`) and are therefore **unexported / package-private**.\n\n#### [BUSINESS_LOGIC]\n## Domain Concepts \u0026 Business Rules\n\n### Core Concept: Recursive Directory Synthesis\nA directory's summary is derived by recursively synthesizing its children (subdirectories) and files together. The result for any module path is a consolidated text representation stored in cache and written to disk under `docs/modules/\u003cpath\u003e`.\n\n### Caching Strategy with Hash-Based Invalidation\n- **Module-level cache**: Stores the final synthesized summary per directory path. Reused when the directory's affected status indicates no changes.\n- **File-level cache**: Stores SHA256 hash + extracted facts. Cache hit is determined by comparing a precomputed or read file's hash against the cached entryβ€”only re-reads on miss.\n\n### Context-Aware Chunking\nBefore synthesis, the available LLM context window size (tokens) is queried. File content limits are calculated as 75% of that window (remaining 25% reserved for prompts/output). Files exceeding this limit are split into overlapping chunks to preserve cross-chunk continuity.\n\n### Multi-Step Extraction Pipeline\nFor each file chunk:\n1. The system prompt is augmented with the current extraction step's prompt.\n2. A single LLM call extracts facts from the chunk, producing one fact per step.\n3. All steps produce their respective facts for the same file, then those facts are consolidated into a single consolidated entry via `reduceFileFacts`.\n\n### Synthesis Order and Assembly\nThe function assembles components in this order:\n1. File facts (in sorted child name order).\n2. Child subdirectory summaries (after all files processed).\n\nThe combined component list is then reduced to a final summary using `reduceInChunks`, which handles the multi-step consolidation at the directory level.\n\n### Affected-Dir Pruning\nIf a directory path has no affected status and a cached module summary exists, it short-circuits: returns the cached value immediately without reprocessing children or files.\n\n#### [STATE_AND_CONCURRENCY]\n**Mutable State:**\n\n| Field | Type | Mutated? | Location |\n|-------|------|----------|----------|\n| `p.cache.Files[f]` | `FileCacheEntry` (in `*MetadataCache`) | Yes β€” written when file facts are computed and cached | Line ~146: `p.cache.Files[f] = FileCacheEntry{...}` |\n| `p.cache.Modules[node.Path]` | string (in `*MetadataCache`) | Yes β€” set to synthesized summary or empty string on miss | Line ~172 \u0026 180: `p.cache.Modules[node.Path] = ...` |\n\n**Concurrency Mechanisms:** None. No `sync.Mutex`, channels, atomic operations, or `context.WithCancel` (for cancellation) are used in this file. The shared mutable state (`*MetadataCache`) is accessed directly by pointer without any locking.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects (External Communication)\n\n### Network / LLM Calls\n- **`p.client.CallLLM(p.ctx, systemPrompt, ...)`** β€” invoked per-extraction-step per-chunk for every processed file that has no cache hit. This is the primary network side-effect: it sends prompts to an external LLM service and receives a response string.\n\n### Disk I/O\n- **Reads:** `tools.ReadFileSafely(p.repoRoot, f)` β€” reads individual source files from disk using the precomputed hash (when available) or on cache miss. Called twice in the worst case per file (once with hash check, once unconditionally if content is needed).\n- **Writes:** `tools.WriteFileSafely(p.repoRoot, modulePath, []byte(finalSum))` β€” persists synthesized summaries to `docsDir/modules/\u003csafe-filename\u003e`. This is a synchronous write; any failure propagates as an error.\n\n### No visible DB operations in this file. All external interaction goes through the LLM caller and disk I/O helpers.\n\n---\n\n## Error Handling Patterns\n\n| Source of Failure | Behavior |\n|---|---|\n| **Context cancellation** (`p.ctx.Err()`) | Checked at function entry and inside per-file loops. Returns context error directly if cancelled mid-processing. |\n| **Recursive child synthesis failure** | Propagated immediately: `return \"\", err` |\n| **File read failure** (cache miss + read fails) | Logged as status event, then the file is **skipped** (`continue`). The loop proceeds to other files. No error returned for that file. |\n| **LLM call failure** during extraction steps | Wrapped with `fmt.Errorf(\"LLM error extracting %s for %s: %w\", step.Name, f, err)` and **returned immediately**. This is a hard stop β€” no further files or children are processed. |\n| **`reduceFileFacts` / `reduceInChunks` failure** | Propagated directly (`return \"\", err`). Hard stop. |\n| **Write failure** after synthesis completes | Wrapped with `fmt.Errorf(\"failed to write module documentation for %s: %w\", node.Path, err)`. Hard stop β€” the synthesized result is lost if this fails. |\n\n### Key Observations\n- The LLM error path is a hard fail (does not skip to other files). The file-read error path is soft (skips one file and continues). These are asymmetric β€” if one file's extraction fails, the entire subtree aborts; if one file can't be read, it's silently skipped.\n- No retry logic is visible for any of these errors within this function.\n- Cache entries are only updated on success paths (after `reduceInChunks` completes), so a failed synthesis does not leave stale cache data behind." + }, + "internal/engine/tree.go": { + "sha256": "21a031a938380004c36613d4e9b002c48fe049dbcff60bd8b9605a7143ecd4c5", + "facts": "#### [API_SIGNATURES]\n## Public Surface Area of `tree.go`\n\n| Symbol | Kind | Description / Fields |\n|---|---|---|\n| `FileChange` | Struct (exported) | `Path string`, `Status string` // \"Added\", \"Modified\", \"Deleted\" |\n| `DirNode` | Struct (exported) | `Path string`, `Files []string`, `Children map[string]*DirNode` |\n\n**Note:** No exported methods or interfaces are present. The file contains three unexported functions (`propagateAffected`, `determineAffected`, `buildTree`) and one internal package import of `security`, which are not part of the public surface.\n\n#### [BUSINESS_LOGIC]\n## Domain Concepts\n\nThis file implements **directory-aware change propagation** β€” determining which directory nodes are affected by a set of file changes across a hierarchical tree structure. The domain concerns:\n\n1. **File Change Tracking**: A `FileChange` struct represents individual path/status pairs (Added, Modified, Deleted).\n2. **Hierarchical Tree Structure**: The system models files and directories as nested nodes (`DirNode`), preserving parent-child relationships based on filesystem paths.\n3. **Affected-Status Propagation**: When a file changes, its status must cascade upward β€” any parent directory containing affected children becomes affected itself.\n\n## Business Rules / Algorithmic Flow\n\nThe logic follows three phases:\n\n**Phase 1 β€” Build the tree.** Given raw file paths, construct a directory tree by splitting each path on `/`, creating intermediate `DirNode` entries for each subdirectory, and attaching files to their leaf nodes.\n\n**Phase 2 β€” Determine affected directories from changes.** For each input change:\n- Mark any directly changed file as affected (if it exists in the current set).\n- When a file is *Deleted*, mark its parent directory as affected.\n\nThen walk every node recursively and check additional conditions that also make a node affected:\n- If the node's path matches a module documentation location (`docs/modules/\u003csafe-name\u003e`), but no such doc currently exists on disk, mark it affected.\n- If the node has an empty cached metadata entry, mark it affected.\n\n**Phase 3 β€” Propagate status upward.** Walk the tree and propagate any `affected` flag from children to parents so that a parent is considered affected if *any* child is.\n\n## High-Level Flow Summary\n\n```\nInput: list of file paths β†’ build directory tree β†’ compute per-node affected flags β†’ propagate flags up the hierarchy β†’ output map[string]bool of affected directory nodes\n```\n\n#### [STATE_AND_CONCURRENCY]\n**Mutable State:**\n\n| Variable / Field | Scope | Mutation Type |\n|---|---|---|\n| `DirNode.Files` (slice) | Struct field, modified by `buildTree` via `append` | Mutated |\n| `DirNode.Children` (map) | Struct field, initialized in `buildTree`, never re-assigned after construction | Not mutated post-init |\n| `affectedDirs` (`map[string]bool`) | Parameter to `propagateAffected` and `determineAffected` | Mutated in-place via `affectedDirs[node.Path] = true`, `affectedDirs[path.Dir(c.Path)] = true` |\n\n**Concurrency Mechanisms:** None. No `sync.Mutex`, channels, atomics, or other synchronization primitives are used anywhere in this file.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects \u0026 Error Handling Analysis β€” `tree.go`\n\n### I/O Operations\n\n| Function | Operation | Direction |\n|----------|-----------|-----------|\n| **determineAffected** | `security.SafeResolve(repoRoot, modulePath)` | External package call (filesystem traversal) |\n| **determineAffected** | `os.Stat(absModulePath)` | Read-only filesystem stat check |\n\nAll other functions (`propagateAffected`, `buildTree`) are pure computations with no I/O.\n\n---\n\n### Error Handling Patterns\n\n#### 1. `determineAffected` β€” Conditional swallow (sentinel-style)\n```go\nabsModulePath, err := security.SafeResolve(repoRoot, modulePath)\nif err == nil {\n if _, err := os.Stat(absModulePath); os.IsNotExist(err) {\n affectedDirs[n.Path] = true\n }\n}\n```\n\n- `security.SafeResolve` error is **swallowed** β€” only the success path is entered. Downstream consequences are unknown; this does not surface to callers.\n- `os.Stat` error is handled: the code specifically checks for `os.IsNotExist(err)` and treats \"missing module\" as a signal that the directory is affected. Other errors (e.g., permission denied) are silently ignored.\n\n#### 2. `propagateAffected` β€” No error handling needed\nPure recursive traversal of `DirNode`; no external calls, no error paths to handle.\n\n#### 3. `buildTree` β€” No error handling\nAssumes all inputs (`files []string`) are valid. No filesystem or OS interaction. If a path is malformed at runtime, it will produce incorrect tree structure silently (no panic, no return)." + }, + "internal/engine/utils.go": { + "sha256": "64a5a8d76570f4f398d694c2ecc7f243c8b07bdf459009a061f463606fda9128", + "facts": "#### [API_SIGNATURES]\n# Public Surface Area: `internal/engine/utils.go`\n\n## Exported Items\n\n### None\n\nThis file defines no exported (capitalized) structs, interfaces, or methods. The two functions declared (`toSafeMarkdownFilename`, `makeLogEvent`) use lowercase identifiers and are therefore unexported. Two capitalized types (`EventType`, `LogEventFunc`) are referenced but not defined in this file.\n\n#### [BUSINESS_LOGIC]\n## Business Rules \u0026 Domain Concepts\n\n### 1. Documentation Generation Pipeline\nThe `toSafeMarkdownFilename` function enforces a naming contract: any module path is transformed into a safe filename by replacing `/` with `_`, then appending `.md`. This supports an implicit rule that engine modules map one-to-one to markdown documentation files, and the root module defaults to `\"root.md\"`.\n\n### 2. Optional Event Notification Pattern\nThe `makeLogEvent` function implements an optional callback adapter for a logging system. It takes a user-provided event handler (`onEvent`) and returns a function that accepts `(EventType, message)` pairs. If no handler is supplied, events are silently swallowed β€” meaning the logging pipeline has no required consumer; it's purely additive instrumentation.\n\n### 3. Event Domain Model (Referenced)\nThe code references an `Event` struct with at least two fields: `Type` (`EventType`) and `Message` (`string`). This indicates a typed event model where events are categorized by type and carry descriptive payloads, suggesting the engine tracks domain occurrences as discrete, typed records.\n\n### High-Level Algorithmic Flow\n1. A module path is converted into a markdown filename via character substitution and empty-value fallback.\n2. A log callback is optionally wired: when invoked with an event type and message, it either forwards to the registered handler or drops the call silently.\n\n#### [STATE_AND_CONCURRENCY]\n**Mutable State:** None found in this file. All functions (`toSafeMarkdownFilename`, `makeLogEvent`) are pure β€” they take inputs and return values without modifying any package-level globals or struct fields.\n\n**Concurrency Mechanisms:** None. No mutexes, channels, atomic operations, or other synchronization primitives are used.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects \u0026 Error Handling Analysis β€” `internal/engine/utils.go`\n\n### I/O Communication\n\n**No external communication.** Both functions are pure in-memory transformations:\n\n- **`toSafeMarkdownFilename`**: Converts a string path to a markdown filename. No network, disk, DB, or file-system interaction occurs.\n- **`makeLogEvent` / closure**: Wraps an `onEvent` callback into a new function that constructs and forwards events. No I/O anywhere in this flow.\n\n### Error Handling\n\n**No error wrapping, no sentinel values, no panics.** Errors are swallowed silently:\n\n1. **`toSafeMarkdownFilename`** returns only the result string β€” it does not return an `error`. If a caller needs to detect invalid input (e.g., characters other than `/` in the path), this function provides no escape hatch; they simply end up in the output filename unchanged.\n2. **`makeLogEvent`** checks whether `onEvent` is nil. If it is, the returned closure does nothing when invoked β€” no panic, no error, just a silent no-op. The caller receives back a valid function that will succeed (by doing nothing) rather than failing loudly.\n\n### Summary\n\n| Concern | Behavior |\n|---|---|\n| External I/O | None |\n| Error propagation | Absent; errors are not returned or wrapped |\n| Silent failure paths | `onEvent == nil` β†’ swallowed without notice |\n| Input safety | Only `/` is replaced; other special characters pass through unmodified |" + }, + "internal/security/errors.go": { + "sha256": "eaf4e252d4b8701cb50b75196bf95b853d9b43b8fd8f8902aa2b1d0c86029916", + "facts": "#### [API_SIGNATURES]\n# Public Surface Area: `errors.go` (Module: internal/security)\n\n## Package-Level Variables\n\n| Identifier | Type | Description |\n|---|---|---|\n| `ErrPathTraversal` | `error` | Returned when a path resolves outside the repository root |\n| `ErrLockHeld` | `error` | Returned when another code-reducer process holds the file lock |\n\n#### [BUSINESS_LOGIC]\n## Domain Concepts\n\nThis file defines **sentinel error types** that represent two security violation categories within the system:\n\n1. **Path Traversal Violation** (`ErrPathTraversal`) β€” signals when a resolved path escapes outside the repository root boundary. This is a classic path traversal / directory escape guardrail.\n2. **Lock Conflict** (`ErrLockHeld`) β€” signals when another process already holds an exclusive file lock, indicating contention on a shared resource.\n\n## High-Level Algorithmic Flow\n\nThis file does not implement any algorithm. It serves as a **shared contract layer**: it provides the canonical error values that callers can match against to branch their business logic. The actual detection and enforcement of these rules live in other files within the `internal/security` module.\n\n#### [STATE_AND_CONCURRENCY]\nNo mutable state detected. Both package-level variables (`ErrPathTraversal`, `ErrLockHeld`) are initialized once to immutable error sentinels via `errors.New` and remain read-only throughout this file's scope. No concurrency mechanisms (e.g., mutexes, channels) are present in this module.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects / External Communication\n\n| Interaction | Evidence |\n|---|---|\n| **Filesystem** | `ErrPathTraversal` comment indicates the package communicates with the filesystem β€” specifically, resolving paths against a repository root boundary. The error fires when a path escapes that boundary (classic path-traversal guard). |\n| **Process coordination / lock mechanism** | `ErrLockHeld` comment indicates the package participates in an external locking protocol. The phrase \"another code-reducer process\" implies inter-process communication, likely via OS file locks or a lockfile on disk. |\n\nNo direct network calls, DB access, or logging are visible within this file itself β€” those interactions belong to other functions in `security` that return these sentinels.\n\n## Error Handling Strategy\n\n| Aspect | Finding |\n|---|---|\n| **Type** | Sentinel errors (pre-allocated variables via `errors.New`). Not wrapped; not panics. |\n| **Detection by caller** | Because they are sentinel values, a caller can use `errors.Is(err, ErrPathTraversal)` or `errors.As` patterns to match them against an error chain. No wrapping means the identity is preserved directly β€” no need for unwrapping logic on this file's side. |\n| **Panic handling** | None visible here. The design returns errors rather than panicking. Callers must handle these errors themselves (wrap, retry with lock release, abort, etc.) β€” this file does not recover or suppress them. |\n| **Error wrapping** | No `fmt.Errorf` wrapping is present in this file; the sentinels are returned as-is by whoever triggers the condition. |\n\n## Summary\n\nThis module exposes two sentinel errors that signal specific external conditions: filesystem boundary violations and lock contention between processes. It communicates with the outside world indirectly β€” other functions in the `security` package perform the actual I/O (path resolution, lock acquisition) and return these sentinels to callers. No panic recovery or error wrapping occurs here; downstream code is expected to handle them explicitly." + }, + "internal/security/security.go": { + "sha256": "5dbfee5adec3701c01c5a4db8f512ef2717579ef9c7a2dbf11f3eb22a63bc1e9", + "facts": "#### [API_SIGNATURES]\n## Public Surface Area of `internal/security` / `security.go`\n\n### Structs\n\n**`SimpleLock`**\n\n- Fields:\n - `lockPath string`\n - `file *os.File`\n - `mu sync.Mutex`\n - `closed bool`\n\n### Methods on `*SimpleLock`\n\n| Method | Signature |\n|--------|-----------|\n| `Unlock()` | `(error)` |\n\n### Functions (Package-level)\n\n| Function | Signature |\n|----------|-----------|\n| `SafeResolve(repoRoot, inputPath string)` | `(string, error)` |\n| `AcquireLock(repoRoot string)` | `(*SimpleLock, error)` |\n| `EnsureGitignoreHasLockfile(repoRoot string)` | `(error)` |\n\n#### [BUSINESS_LOGIC]\n## Domain Concepts \u0026 Business Rules\n\n### 1. Path Traversal Prevention\n`SafeResolve` implements a path containment rule: any input path must resolve strictly inside the repository root. It achieves this by:\n- Resolving symlinks on the existing ancestor to prevent symlink-based escape vectors\n- Walking up from the target until it finds a physically existing directory, then rejoining with the remaining suffix components\n- Verifying the final result is under `resolvedRoot` via relative path check\n\n### 2. Process Mutual Exclusion (File Locking)\nThe `SimpleLock` type enforces single-process execution:\n- **Acquire** uses `O_EXCL` flag on file creation for atomic lock acquisition β€” if the lockfile already exists, the operation fails immediately rather than silently succeeding\n- The lockfile stores the PID of the owning process, enabling identification and cleanup of stale locks\n- **Unlock** is idempotent: closing the file descriptor and removing the lockfile in a thread-safe manner\n\n### 3. Gitignore Maintenance Rule\n`EnsureGitignoreHasLockfile` implements a convention: the lockfile must be excluded from version control but automatically tracked if missing. It:\n- Appends the lockfile entry to `.gitignore` only if not already present (idempotent)\n- Uses atomic rename via temp file + sync before close to prevent partial writes\n\n### High-Level Algorithmic Flow\n\n```\n[Initialize] β†’ [Ensure path safety for all repo-root operations] β†’ \n[If no process holds the lock] β†’ [Create lockfile with PID] β†’ \n[Run business logic within locked session] β†’ \n[Release lock (close file + remove lockfile)] β†’ \n[Append entry to .gitignore if needed]\n```\n\nThe two non-obvious constraints are: (1) path resolution walks symlinked ancestors before rejoining suffix components, and (2) gitignore updates use atomic rename with explicit sync before close.\n\n#### [STATE_AND_CONCURRENCY]\n**Mutable State:**\n\n| Variable | Type | Mutated? | Protected By |\n|----------|------|----------|--------------|\n| `SimpleLock.mu` (sync.Mutex) | struct field on receiver | Yes – Lock() and Unlock() acquire/release it; `l.closed = true`, `l.file = nil` happen inside the critical section | `sync.Mutex` |\n\n**Concurrency Mechanisms:**\n\n1. **File lock via O_EXCL**: `AcquireLock` uses `os.O_EXCL` to atomically create the lock file, preventing concurrent acquisition from different processes (cross-process mutual exclusion at the filesystem level).\n2. **`sync.Mutex` on `SimpleLock`**: Protects internal struct fields (`closed`, `file`) during `Unlock()`.\n\n**No global package-level mutable state** exists in this file β€” all state is either local to a function or encapsulated within the `SimpleLock` receiver, which is accessed only through its methods.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects \u0026 Error Handling Analysis: `internal/security/security.go`\n\n### 1. `SafeResolve(repoRoot, inputPath)` β€” Disk I/O, No Network/DB\n\n**Side effects:**\n- Calls `filepath.Abs`, `EvalSymlinks` (resolves symlinks on existing ancestor path)\n- Performs repeated `os.Lstat` walks up the directory tree until a physically-existing ancestor is found\n- Calls `EvalSymlinks` again on that ancestor\n- Calls `filepath.Rel` to verify containment within repo root\n\n**Error handling:**\n| Condition | Handling |\n|---|---|\n| Any error from `Abs`, first `EvalSymlinks`, second `EvalSymlinks`, `Lstat` (non-exist), `Rel` | Wrapped with descriptive message using `fmt.Errorf(..., err)` β€” wraps the underlying cause via `%w` where applicable |\n| Unrecognized error from `Lstat` (`!os.IsNotExist`) | Returned immediately as wrapped error |\n| Path escapes repo root (`..` prefix or non-nil rel) | Returns sentinel-style error wrapping `ErrPathTraversal` with the original input path |\n\n**Note:** All errors returned to caller are typed via `%w`, making them recoverable by a wrapping check. No panic, no silent swallow.\n\n---\n\n### 2. `SimpleLock.Unlock()` β€” Disk I/O (file close + remove), Thread-safe\n\n**Side effects:**\n- Closes the lock file descriptor (`os.File.Close`)\n- Calls `os.Remove` on the lock path\n\n**Error handling:**\n| Condition | Handling |\n|---|---|\n| Lock already closed (`l.closed == true`) | Returns `nil` silently β€” this is an idempotent design choice, not a bug |\n| Close fails (`l.file.Close() != nil`) | Captured in local `err`; file set to `nil` so subsequent close is no-op |\n| Remove fails AND error exists from Close | If remove returns non-nil and not `os.IsNotExist`, the remove error **overwrites** any prior close error because of this condition: `if err == nil { err = removeErr }`. So if close failed but remove succeeded, the remove error is reported. If close failed first, remove error is ignored. |\n| Remove fails with `os.IsNotExist` (file already gone) | Silently accepted β€” treated as normal idempotent cleanup |\n\n**Note:** The overwrite logic (`if err == nil { err = removeErr }`) means the final returned error depends on which operation *last* failed, not necessarily the most severe one. This is an intentional design choice but worth flagging: if close fails and remove succeeds (or vice versa), only one error path dominates.\n\n---\n\n### 3. `AcquireLock(repoRoot)` β€” Disk I/O (create + write lockfile)\n\n**Side effects:**\n- Resolves `.code-reducer.lock` via `SafeResolve`\n- Opens file with `O_WRONLY|O_CREATE|O_EXCL, defaultFilePerm` (atomic exclusive creation)\n- Writes PID to the newly created lockfile\n- On failure: closes file, removes lockfile\n\n**Error handling:**\n| Condition | Handling |\n|---|---|\n| `SafeResolve` fails | Propagated as-is to caller |\n| File open fails AND error is `os.IsExist` (lock already held) | Returns wrapped sentinel error wrapping `ErrLockHeld`, including the lock path β€” caller can detect stale-lock condition specifically |\n| File open fails with any other error | Wrapped and returned: `\"failed to acquire lock at %s: \u003ccause\u003e\"` |\n| Write PID fails | File is closed, lockfile removed (cleanup), wrapped error returned |\n\n**Note:** `O_EXCL` provides atomicity β€” no race between existence check and open. The sentinel error for \"lock held\" uses `%w`, so a caller can detect this via `errors.Is(err, ErrLockHeld)`.\n\n---\n\n### 4. `EnsureGitignoreHasLockfile(repoRoot)` β€” Disk I/O (read + write .gitignore), Atomic rename pattern\n\n**Side effects:**\n- Resolves `.gitignore` via `SafeResolve`\n- Reads existing `.gitignore` content (`os.ReadFile`)\n- Parses lines, checks if lockfile entry already exists\n- If not present: constructs new content with the lockfile entry appended (with trailing-newline normalization)\n- Creates a temp file in same directory as `.gitignore`\n- Writes to temp file, calls `Sync()` on it, closes it, then `Chmod`s it\n- Renames temp file over original `.gitignore`\n\n**Error handling:**\n| Condition | Handling |\n|---|---|\n| `SafeResolve` fails | Propagated as-is |\n| Read fails AND not `os.IsNotExist` (file exists but read error) | Wrapped: `\"error reading .gitignore: \u003ccause\u003e\"` β€” caller must handle missing-file vs I/O-error distinction separately |\n| CreateTemp fails | Wrapped with descriptive message including `%w` |\n| Write to temp file fails | Wrapped; temp file is NOT cleaned up (defer handles close + remove, but write error returns before cleanup path executes) |\n| Sync fails | Wrapped |\n| Close fails | Wrapped β€” note that `os.Chmod` and subsequent rename may have already run on a partially-written temp file if this error occurs late in the sequence |\n| Chmod fails | Wrapped |\n| Rename (atomic swap) fails | Wrapped with descriptive message |\n\n**Note:** The deferred function (`tmpFile.Close()` + `os.Remove(tmpName)`) runs regardless of return path. However, because the function returns *before* the defer has a chance to execute in some failure paths (e.g., write fail), cleanup may not happen for intermediate states β€” but since we're using temp file with unique name and same-directory rename, the OS-level temp file is cleaned by `os.Remove` in the defer on any error return path. Still worth noting: if *write* fails mid-stream before close/rename, the deferred Remove will clean up the partially-written temp file.\n\n---\n\n### Summary Table\n\n| Function | I/O Type | Sentinels / Wraps | Panics? | Silent Swallows? |\n|---|---|---|---|---|\n| `SafeResolve` | Filesystem (stat, symlink resolve) | Yes β€” wraps with `%w`; sentinel for path traversal via custom `ErrPathTraversal` | No | No |\n| `SimpleLock.Unlock()` | File close + remove | Partial β€” idempotent-close returns nil; remove errors overwrite prior close errors only if both fail | No | Yes (already-closed case) |\n| `AcquireLock` | Create + write lockfile | Yes β€” sentinel for \"lock held\" via custom `ErrLockHeld`; wraps other failures with `%w` | No | No |\n| `EnsureGitignoreHasLockfile` | Read + atomic rewrite of `.gitignore` | Partial β€” missing-file (`os.IsNotExist`) is treated as non-error; all other errors wrapped with `%w` | No | Yes (missing .gitignore case returns nil) |\n\n### Key Patterns Observed\n\n1. **Error wrapping throughout** β€” All functions use `fmt.Errorf(..., err)` with `%w`, making root-cause recovery possible via `errors.Is()`.\n2. **Custom sentinel errors** β€” At least two custom error variables referenced: `ErrPathTraversal` and `ErrLockHeld`. These enable typed error detection by callers.\n3. **No panic anywhere** β€” All failure paths return to caller with an error value.\n4. **Idempotent operations deliberately silent** β€” `Unlock()` on already-closed lock returns nil; missing `.gitignore` in EnsureGitignoreHasLockfile returns nil. These are design choices, not accidentals.\n5. **Atomic rename pattern for .gitignore** β€” Write to temp β†’ Sync β†’ Close β†’ Chmod β†’ Rename is the standard atomic-update pattern. The `Sync()` call before close is explicit and unusual (normally deferred), but here it's called eagerly before close in this implementation." + }, + "internal/tools/file_tools.go": { + "sha256": "f2d2fe4bea860a28a18424bdf8d7d0af16e935fdda0ba345d8b881f9540e20a0", + "facts": "#### [API_SIGNATURES]\n# Public Surface Area: `internal/tools/file_tools.go`\n\n## Exported Functions\n\n| Function | Input Types | Output Types |\n|---|---|---|\n| `ReadFileSafely(repoRoot, virtualPath string)` | `string`, `string` | `[]byte`, `error` |\n| `WriteFileSafely(repoRoot, virtualPath string, content []byte)` | `string`, `string`, `[]byte` | `error` |\n| `LoadGitignore(repoRoot string)` | `string` | `[]string`, `error` |\n| `ShouldIgnoreFile(repoRoot, relPath string, gitIgnore *ignore.GitIgnore)` | `string`, `string`, `*ignore.GitIgnore` | `bool` |\n| `DiscoverCodeFiles(repoRoot string, ignores []string)` | `string`, `[]string` | `[]string`, `error` |\n| `IsBinaryFile(path string)` | `string` | `bool` |\n\n## Exported Constants (none)\n\nNo exported constants are declared in this file. All constants (`binaryDetectionBufSize`, `defaultDirPerm`, `defaultFilePerm`) use lowercase identifiers and remain package-private.\n\n## Exported Structs / Interfaces (none)\n\nNone.\n\n#### [BUSINESS_LOGIC]\n## Domain Concepts\n\nThis module implements **safe file I/O operations** and **intelligent source-code discovery** for a repository-scanning tool. It solves three domain problems:\n\n1. **TOCTOU race-condition prevention** β€” File read/write paths use atomic patterns (temp-file + rename) to prevent symlink/swap attacks between check and use windows.\n2. **Selective file discovery** β€” Walks the entire codebase but filters out ignored directories, hidden files, build artifacts, dependency trees, binary outputs, and non-source languages.\n3. **Binary-vs-text classification** β€” Distinguishes text source files from binary content by scanning for null bytes, treating binaries as uninteresting during discovery.\n\n## High-Level Algorithmic Flow\n\n### Safe File Operations (Read/Write)\n\nBoth `ReadFileSafely` and `WriteFileSafely` follow the same TOCTOU-safe sequence:\n1. Resolve a virtual path to an absolute filesystem path using `SafeResolve`.\n2. If writing, create parent directories with default permissions.\n3. Create a temporary file in the target directory.\n4. Write content to the temp file and sync/close it before renaming into place.\n5. On read paths, verify that the resolved target is not a symlink (both pre-open and post-open checks) so a TOCTOU swap cannot substitute a real file with a symlink.\n\n### File Discovery Pipeline (`DiscoverCodeFiles`)\n\n1. Compile user-supplied ignore patterns + `.gitignore` entries into a single `GitIgnore` object.\n2. Walk the entire directory tree recursively. For each entry:\n - If it is a directory, check whether its name starts with `.` or matches any compiled ignore pattern; if so, skip the subtree entirely.\n - If it is a file, delegate to `ShouldIgnoreFile`.\n3. `ShouldIgnoreFile` applies layered rules in order:\n - First checks against the compiled gitignore patterns (user + `.gitignore`).\n - Then rejects any path component that begins with `.` or ends with `.egg-info`, regardless of other rules.\n - If the file extension matches a known text-source language list, it is accepted as code.\n - For unknown extensions, it resolves the absolute path and classifies it as binary if null bytes are detected in the first 1024 bytes; binaries return `true` (ignored).\n4. Files passing all filters are collected into the result slice.\n\n### Result\n\nThe caller receives a list of relative source-code file paths with no build artifacts, hidden directories, dependency trees, or binary outputs included.\n\n#### [STATE_AND_CONCURRENCY]\n**Mutable State:** No mutable state detected in this file. All constants are `const` (compile-time), all variables (`safePath`, `dir`, `tmpFile`, `tmpName`, `patterns`, `files`, `buf`, etc.) are function-local, and no package-level global state is modified.\n\n**Concurrency Mechanisms:** No concurrency mechanisms detected in this file. There are no `sync.Mutex`, channels, `atomic` types, or other synchronization primitives used to protect any shared stateβ€”because there is no shared mutable state to protect.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects (I/O with Outside World)\n\n| Function | External I/O / System Call | Notes |\n|---|---|---|\n| **ReadFileSafely** | `os.Open`, `os.Lstat`, `f.Stat`, `io.ReadAll` | Reads a file. No network, DB, or disk-write operations. |\n| **WriteFileSafely** | `os.MkdirAll`, `os.CreateTemp`, `tmpFile.Write`, `tmpFile.Sync`, `tmpFile.Close`, `os.Chmod`, `os.Rename` + deferred `os.Remove` | Writes to filesystem only. Creates temp file in same dir and renames into final path (atomic-ish). |\n| **LoadGitignore** | `os.Open` (reads `.gitignore`) | Reads one file. If missing, returns `nil` patterns β€” no error raised. |\n| **ShouldIgnoreFile** | `security.SafeResolve`, `IsBinaryFile` β†’ `os.Open`, reads first 1024 bytes | The binary check opens and scans the file on disk every time this function is called β€” a real I/O side effect per call. |\n| **DiscoverCodeFiles** | `filepath.WalkDir` (walks entire tree) + `fmt.Fprintf(os.Stderr, …)` | Walk traverses all directories/files under `repoRoot`. On walk errors it prints to stderr and skips that entry. No network/DB calls. |\n| **IsBinaryFile** | `os.Open`, reads 1024 bytes for null-byte scan | Opens the file on every invocation. Buffered reader of size 1024. |\n\n## Error Handling / Return Patterns\n\n| Function | Approach | Details |\n|---|---|---|\n| **ReadFileSafely** | Wrap + sentinel | Returns `nil, err` for failures; returns a wrapped error with context on each step (`failed to open file`, `failed to lstat`, etc.). Symlink detection returns an explicit \"security violation\" error. |\n| **WriteFileSafely** | Wrap only (no sentinel) | All failure paths return a wrapped error. Directory creation, temp write, sync, close, chmod, and rename failures each get their own context string. No panic usage. |\n| **LoadGitignore** | Graceful for missing file, wrap otherwise | `os.IsNotExist` β†’ returns `nil, nil`. Any other open/read error is returned as-is (no wrapping). |\n| **ShouldIgnoreFile** | Boolean return; swallow walk errors in caller | Returns `true` on any path-resolve failure from `SafeResolve` (treated as \"ignored\"). No error field β€” caller must decide what a false-positive means. |\n| **DiscoverCodeFiles** | Swallow per-entry, propagate final err | Walk callback: if walk returns an error for a single entry, prints to stderr and skips that item (`return nil`). The overall `err` from `WalkDir` is returned at the end β€” so a terminal fatal walk error propagates up. Non-fatal entries are skipped silently. |\n| **IsBinaryFile** | Fail-closed (returns true on any error) | If open fails or read encounters an error other than EOF, returns `true` (assumes binary). This is intentional but means I/O errors are not reported as failures β€” they're treated as \"likely binary\". |\n\n## Summary of Patterns Observed\n\n- **No network calls** anywhere in this file.\n- **No database operations.**\n- **stderr logging** only inside `DiscoverCodeFiles` for walk errors; all other functions return errors instead of logging.\n- **Deferred cleanup** used in `WriteFileSafely` (close + remove temp) and `ReadFileSafely` (defer `f.Close()`).\n- **\"Fail-closed\" assumption**: `IsBinaryFile` treats any I/O error as \"binary\" rather than propagating the actual error.\n- **No panics** observed β€” errors are returned to callers throughout." + }, + "internal/tools/git_tools.go": { + "sha256": "ee9d244fdd8ca3a1dfcc7df969b02a4d68d65c2fe3aba8203d62819bed4efcd0", + "facts": "#### [API_SIGNATURES]\n# Public Surface Area: `git_tools.go` β€” Package `internal/tools`\n\n| Name | Kind | Input | Output |\n|------|------|-------|--------|\n| [RunGit](file:///home/user/projects/go/src/internal/tools/git_tools.go) | Function | `(repoRoot string, args ...string)` | `(string, error)` |\n| [VerifyGitRepo](file:///home/user/projects/go/src/internal/tools/git_tools.go) | Function | `(repoRoot string)` | `error` |\n\n#### [BUSINESS_LOGIC]\n**Domain Concepts:**\n* Git repository validation: Determining whether a specific directory path corresponds to a functional git work-tree.\n* External command execution within a context: Running shell-like operations against the `git` binary scoped to a target root directory, suppressing pager output.\n\n**Business Rules:**\n1. **Isolation of Environment:** Any operation invoking this tool must specify a repository root; all subprocesses run with that path set as their working directory.\n2. **Failure Handling:** If an external command fails, the returned error message is constructed from standard error (stderr) content rather than being silently dropped.\n\n**High-Level Algorithmic Flow:**\n1. **Execute Git Command:** Initialize a subprocess targeting the `git` binary with the `--no-pager` flag and any additional arguments provided by the caller. Configure this process to execute within the specified `repoRoot`. Capture both standard output and standard error streams. Run the command. If execution returns an error, return the trimmed stderr content wrapped in a descriptive failure message. If successful, return the trimmed stdout content.\n2. **Verify Repository:** Invoke the specific git subcommand (`rev-parse --is-inside-work-tree`) using the general execution logic above. If this invocation results in an error, conclude that the directory is not a valid git repository and return a corresponding error message indicating it is either not a git repository or not inside any parent work-tree.\n\n#### [STATE_AND_CONCURRENCY]\n**Mutable State:** No mutable state. All variables (function parameters, local variables like `cmd`, `stdout`, `stderr`, `trimmedOut`, `trimmedErr`) are scoped to individual functions and never escape their defining scope.\n\n**Concurrency Mechanisms:** None.\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects \u0026 External Communication\n\n| Function | External Process / I/O | Notes |\n|---|---|---|\n| `RunGit` | Spawns `git --no-pager` via `exec.Command`. Captures stdout/stderr into memory (`bytes.Buffer`). No file, network, or DB I/O. | All communication is local process execution. Stdout/stderr are in-memory buffers β€” no disk writes. |\n| `VerifyGitRepo` | Delegates to `RunGit`, so same external git invocation. | Indirect side effect; inherits the subprocess spawn from `RunGit`. |\n\n## Error Handling\n\n- **No panics.** All errors are returned to callers as Go values.\n- **`RunGit`:** On failure, wraps the original error with stderr content: \n `fmt.Errorf(\"git command failed: %v, stderr: %s\", err, trimmedErr)`. Returns `(string, error)` β€” partial stdout is still delivered alongside the error.\n- **`VerifyGitRepo`:** Wraps any error from `RunGit` into a fixed message: \n `\"not a git repository (or any of the parent directories)\"`. The original stderr content is discarded in this wrapper layer.\n\n## Summary\n\n- **I/O surface:** subprocess execution only β€” no network, disk, DB, or other external services.\n- **Error strategy:** wrap-and-return. `RunGit` preserves diagnostic detail; `VerifyGitRepo` normalizes to a single sentinel-style message." + }, + "main.go": { + "sha256": "16ab362b1830374f0306a1990dba8d51da887c1ad126f311c03d242933d2a516", + "facts": "#### [API_SIGNATURES]\n## Public Surface Area of `main.go`\n\n**Package:** `main` (no public exports β€” all identifiers are unexported)\n\n| Item | Type | Notes |\n|------|------|-------|\n| `func main()` | method | Unexported (`m` is lowercase). Entry point for the binary. |\n\n\u003e **Note:** This file contains no exported structs, interfaces, or methods. The only function defined is `main`, which is unexported (starts with a lowercase letter). The call to `cmd.RootCmd.Execute()` references an external package's type and method, not declarations in this file.\n\n#### [BUSINESS_LOGIC]\n### Business Rules / Domain Concepts: None\n\nThis file contains no business logic, domain concepts, or algorithmic flow. It is purely an application bootstrap entry point that delegates all work to an external command package.\n\n### High-Level Algorithmic Flow:\n\n1. **Initialize** the `main` function as Go's program entry point\n2. **Invoke** `cmd.RootCmd.Execute()` β€” this is where all actual logic lives (the file does not define it)\n3. **Handle failure**: if execution returns an error, print a formatted message to `stderr` and exit with code 1\n\n### Summary:\n\nThis is a thin wrapper / CLI entry point for the `code-reducer` application. The real domain behavior β€” parsing arguments, running reduction commands, handling files β€” resides in `cmd/` package imports, which are not present in this file.\n\n#### [STATE_AND_CONCURRENCY]\nNo mutable state\n\n#### [ERRORS_AND_SIDE_EFFECTS]\n## Side Effects (I/O)\n\n- **`fmt.Fprintf(os.Stderr, ...)`** β€” writes a formatted error message to *stderr* when `cmd.RootCmd.Execute()` returns an error. This is stdout-independent output; no disk/network/DB I/O in this file.\n- All other side effects (disk write, network call, DB query) are delegated to the third-party package (`github.com/arrase/code-reducer/cmd`). No direct I/O calls exist here.\n\n## Error Handling\n\n| Mechanism | Behavior |\n|---|---|\n| **Return value** | `cmd.RootCmd.Execute()` returns an `error` (or nil). The returned error is treated as a sentinel-style signal: if non-nil, execution stops. |\n| **Error wrapping** | None visible in this file. Errors are passed through from the cmd package to main with no wrap/unwrap logic. |\n| **Sentinel / panic** | No panic recovery (`recover`) and no custom sentinel types. The code uses raw `error` values only. |\n| **Exit behavior** | On error: prints to stderr, then calls `os.Exit(1)` β€” immediate process termination, no deferred cleanup or graceful shutdown path visible here. |\n\n## Summary\n\nThis file is a thin CLI entrypoint that delegates all work and I/O to the imported `cmd` package. Its observable surface is: **read an error from Execute() β†’ print it to stderr β†’ exit with code 1**." + } + }, + "modules": { + ".": "# Code Reducer β€” Architecture Documentation\n\n## System Overview\n\nCode Reducer is a multi-phase LLM analysis pipeline that generates technical documentation from Go source code. The system operates across four subsystems: **config**, **engine**, **security**, and **tools**. No shared mutable state exists across subsystem boundariesβ€”each module operates on caller-provided arguments and returns results without synchronization primitives.\n\n**Data flow**: `internal/config` resolves a fully-populated configuration via multi-source priority merging β†’ `internal/engine` consumes that config to drive an LLM pipeline β†’ `internal/security` enforces path containment for all filesystem operations the engine performs β†’ `internal/tools` provides file I/O and git discovery primitives used by both security and engine subsystems.\n\n## Entry Point Architecture\n\nThe binary entry point (`main.go`) is a thin wrapper delegating all work to an external command package. Its observable surface: read an error from Execute() β†’ print it to stderr β†’ exit with code 1. The `cmd` module defines the root cobra command, registers subcommands (init, update, setup), resolves configuration from merged sources, routes engine events to stdout/stderr, handles OS signals for graceful shutdown, and delegates all substantive work to an unshown executeCommand entry point.\n\n**Signal handling**: A handler captures INT (os.Interrupt) and TERM (syscall.SIGTERM) via signal.NotifyContext. A deferred call ensures context cancellation when a signal arrives. The engine runner's returned error is then wrapped as `\"documentation run failed: …\"`.\n\n## Configuration Lifecycle\n\n### Types \u0026 Defaults\n\n`config.go` declares three exported types: `ExtractionStep` (per-phase identifiers carrying name string and prompt text), `Config` (aggregate configuration container referenced by all exported functions), and compile-time constants for environment variable keys, Ollama default values (`OllamaDefaultBaseURL = \"http://localhost:11434\"`, `OllamaDefaultModelID = \"ornith:9b\"`, `OllamaDefaultNumCtx = 8192`), file-system defaults (`DefaultDocsDir = \"wiki\"`, `ConfigFileName = \".code-reducer.yaml\"`), and multiline prompt templates.\n\n### File Lifecycle Implementation\n\n`io.go` provides three exported functions: `ConfigExists(cwd string)` returns a boolean, `LoadConfig(cwd string)` returns `(*Config, error)`, and `SaveConfig(cwd string, cfg *Config)` returns an error.\n\n**Atomic Write Pattern for SaveConfig**: The save operation does not write directly to the target path. Sequence: marshal `cfg` into YAML bytes β†’ apply formatting normalization (collapsing single blank lines to double blank lines before known section headers: `system_prompt`, `module_synthesis_prompt`, `architecture_prompt`, `file_fact_consolidation_prompt`, `extraction_steps`, `ignore`) β†’ create a temp file matching `.tmp.*` in the same directory β†’ write normalized YAML into it β†’ sync and close β†’ set permissions via `os.Chmod` β†’ atomically rename temp over original. Readers never observe a partially-written config during save because the old file remains intact until the new one is fully ready to replace it. A deferred function calls `tmpFile.Close()` and `os.Remove(tmpName)`. On success, `Close` runs twice on the temp fileβ€”Go allows this safelyβ€”and the defer's `Remove` returns an error that is silently swallowed because the original config now holds the renamed data.\n\n**Load-Save Roundtrip Integrity**: Loading preserves whatever formatting was originally written; normalization is applied only on write, not read. This decouples the in-memory representation from on-disk formatting expectations. `LoadConfig` returns a fresh `*Config` via unmarshaling with no shared state. Errors from `os.ReadFile` propagate directly; YAML parse failures are wrapped with a `\"failed to parse yaml config\"` prefix chained via `%w`; marshal failures similarly wrap a descriptive message.\n\n### Multi-Source Configuration Resolution\n\n`resolve.go` exposes a single exported function: `ResolveConfig(repoRoot, modelIDFlag, numCtxFlag string)` returning `(*config.Config, error)`. Internal helper `mergeAndDeduplicate` is lowercase and not part of the public surface. Only usage (not definition) of package-level types (`Config`, `OllamaDefaultModelID`, `CodeReducerModelIDEnvKey`, `DefaultExtractionSteps`) is visible here.\n\n**Resolution Priority System**: Each configurable field follows its own independent priority chain, lowest to highest: **defaults β†’ YAML config file β†’ environment variables β†’ CLI flags**. The merge-and-deduplicate logic operates on local function scopes with no cross-goroutine visibility and no shared mutable state.\n\n**Field-Specific Priority Chains**: `ModelID` applies Default \u003e YAML \u003e Env Var \u003e CLI Flag. `OllamaBaseURL` applies Default \u003e YAML \u003e Env Var. `OllamaNumCtx` (context size) applies Default \u003e YAML \u003e Env Var \u003e CLI Flag. `DocsDir` applies Default \u003e YAML. System prompts (`SystemPrompt`, `ModuleSynthesisPrompt`, `ArchitecturePrompt`, `FileFactConsolidationPrompt`) all start with default, then override with YAML if non-empty.\n\n**Validation Rules**: Numeric fields: CLI flag and env var values are parsed as integers; a value must be greater than zero to take effect. Invalid or non-positive values fall through silently. String fields: empty string is treated as \"not set\" β€” the lower-priority default remains active. Config file absence (`config.yml`): not an error; if missing, an empty `Config{}` struct is created and defaults are applied. Other parse errors are surfaced.\n\n**Algorithmic Flow for ResolveConfig**: (1) Load YAML config β€” if it exists and parses successfully; otherwise start with an empty struct (missing-file case). (2) Resolve extraction steps β€” use YAML value if present, else fall back to `DefaultExtractionSteps`. (3) Deduplicate ignore list from the loaded config using a map-based seen-set while preserving first-seen order. (4) For each field, apply its specific priority chain: start with built-in default constant β†’ override with YAML (if non-empty) β†’ override with environment variable (if set and valid for numeric types) β†’ override with CLI flag (highest priority; numeric types require successful integer parse and positive value). (5) Return the fully resolved `Config` struct, or an error if the YAML file fails to parse (excluding missing-file case).\n\n**Error Handling in ResolveConfig**: When `os.IsNotExist(err)` is true during config loading, defaults apply with no error returned. Any other error (parse failure, permission denied) is wrapped with a prefix and returned as `(*Config, nil, error)` β€” the config pointer is `nil`. Silent branches: string comparisons (`!= \"\"`, `\u003e 0`) short-circuit without errors. `strconv.Atoi` failures are caught inline; env/flag values simply not used. Map lookups for deduplication have no error path.\n\n**Always-On Return**: The function always returns a non-nil tuple `(*Config, error)` β€” either a populated config with `nil` error, or a `nil` config with an error string. No panics anywhere in this file.\n\n## Engine Pipeline Architecture\n\n### Constants \u0026 Types\n\n`constants.go` declares compile-time constants: `defaultHTTPTimeout` (10 min), `maxErrorBodyBytes` (1024 bytes), `defaultChunkOverlap` (800 chars), `minNumCtxFloor` (512 tokens), `contextWindowAllocRatio` (0.75), `maxCharsMultiplier` (3), `metadataFileName` (`.metadata.json`), `agentsFileName` (`AGENTS.md`), `defaultDirPerm` (0755). Type alias `LogEventFunc` carries signature `func(EventType, string)` with no business rules defined here.\n\n### LLM Client Communication\n\nUnexported except `Message` struct (`Role string`, `Content string`). The package-private `llmClient` carries: model ID, base URL, context count, HTTP client pointer. All set once during construction; never modified afterward within this file. No mutexes or channels used.\n\n**Algorithmic Flow**: (1) Prepare payload: assemble request body by prepending `systemPrompt` as a system-role message before any user-supplied messages, serialize as JSON to Ollama `/api/chat`. (2) Send HTTP request: single synchronous POST with fixed timeout; no retries (fail-fast policy). (3) Handle response: on 200 OK β†’ deserialize body β†’ extract `message.content` field. Non-2xx status code β†’ error string containing both HTTP status and raw response payload. (4) Return result: extracted text content or wrapped error describing what went wrong.\n\n**Error Handling Patterns**: Marshal failure wraps via `%w` for caller inspection via `errors.Is`. HTTP request construction returns unwrapped; context lost for callers. Network/transport error returns unwrapped; timeout vs DNS vs connect indistinguishable to caller. Success-path body read returns unwrapped (rare). JSON unmarshal failure wraps via `%w`. Non-OK status response uses custom message discarding read error (`_`); uses undefined `maxErrorBodyBytes` constant. No retry logic.\n\n### Runner \u0026 Entry Point\n\n`Runner` wraps a config pointer and exposes a single `Run(ctx, repoRoot, mode, onEvent)` method that is the only public entry point into the pipeline. It performs four sequential operations: (1) Gitignore lockfile maintenance β€” attempts to append the lockfile path to `.gitignore`. Failure is logged via `onEvent(EventStatus, ...)` and execution continues; no error returned. (2) Repository lock acquisition β€” calls `security.AcquireLock(repoRoot)`. On failure, returns wrapped error: `\"failed to acquire repository lock: %w\"`. No partial work proceeds without the lock. (3) Mode dispatch β€” constructs an LLM client from config fields (`cfg.ModelID`, `cfg.OllamaBaseURL`, `cfg.OllamaNumCtx`) and instantiates an `orchestrator`. Dispatches to either `RunInit` or `RunUpdate`; any other mode returns `\"unsupported mode: %s\"`. (4) Deferred lock release β€” `defer lock.Unlock()` runs after the pipeline completes, regardless of success or failure.\n\nThe `Runner.cfg` field is set once in `NewRunner(cfg)` and never mutated within this file. The pointer type allows external reassignment but no such mutation occurs here.\n\n### Orchestrator \u0026 Pipeline Modes\n\nUnexported struct with two exported constants on the `EventType` type alias: `EventStatus = \"status\"` and `EventError = \"error\"`. The unexported `orchestrator` struct carries a method receiver; its public surface is empty.\n\n**RunInit β€” Cold Start Flow**: (1) setupPipeline() β†’ discover files, compute hashes, load .gitignore (swallowed), load cache (swallowed). (2) Mark every directory as affected (full regeneration). (3) Synthesize root summary from full tree via LLM calls. (4) Generate architecture.md + quickstart.md using root summary. (5) Write agent guidelines file (append if marker missing, overwrite otherwise). (6) Persist cache to disk β†’ complete.\n\n**RunUpdate β€” Incremental Refresh Flow**: (1) setupPipeline() β†’ discover current files, compute hashes. (2) If extraction steps changed β†’ invalidate entire cache (.StepsHash mismatch resets Files and Modules maps). (3) Classify all discovered files into Added/Modified/Deleted buckets by comparing current vs cached SHA-256 hashes. (4) Build directory tree from allowed (existing + new) files only. (5) Purge stale module summaries: for any `cache.Modules` path not represented in the live tree, delete both the cache entry and the physical `.md` file (`_ = os.Remove(...)` β€” error swallowed). (6) Compute affected directories; propagate upward through parent dirs. (7) If `len(affectedDirs) == 0` β†’ return early (no-op, docs are up to date). (8) Synthesize root summary from the affected region via LLM calls. (9) Re-generate architecture.md + quickstart.md only if `.` is in `affectedDirs`, or either file does not exist on disk, or resolution fails (treated as \"not exists\"). (10) Persist cache β†’ complete.\n\n**Pipeline Context**: Unexported struct carrying: LLM client pointer, repo root string, config pointer, `*MetadataCache` pointer, affected directory set map, precomputed file hashes map, and event logger callback. Constructed fresh per invocation; no shared instances exist across goroutines in this file.\n\n### Cache Persistence Layer\n\nTwo structs with no methods: **`FileCacheEntry`** β€” pairs a SHA256 digest string (`sha256`) with extracted facts string (`facts`). Stored in `MetadataCache.Files map[string]FileCacheEntry`. **`MetadataCache`** β€” carries version int, steps hash string, files map, and modules map (module dir β†’ synthesized summary).\n\n**IsInitialized(repoRoot, docsDir) bool**: Returns true only if the cache file exists AND is readable in one call. Reuses the existence check from `loadMetadataCache`. No error return.\n\n**saveMetadataCache(repoRoot, docsDir, cache) error**: Serializes to `{docsDir}/{metadataFileName}` with indented JSON (`json.MarshalIndent(cache, \"\", \" \")`). Marshaling errors propagate directly; write errors pass through `tools.WriteFileSafely`. The version field is pinned to the current value on every save.\n\n**Version Compatibility Guard**: On load: if `cache.Version != currentCacheVersion` (currently 1), returns a fresh zero-valued `MetadataCache{}` with nil error. No warning, no panicβ€”caller sees success with an empty cache. This prevents stale metadata from corrupting future runs without explicit migration handling.\n\n**Fault-Tolerant Initialization**: If the file does not exist (`os.ErrNotExist` sentinel matched via `errors.Is`), returns a populated zero-valued cache with nil error. If it exists but cannot be read or unmarshaled, returns wrapped error `\"failed to read metadata cache: %w\"`. Callers can distinguish \"not yet initialized\" from \"corrupt data.\"\n\n### Chunking \u0026 Reduction Engine\n\nAll unexported; package-private. Implements a **context-aware recursive reduction** for LLM-based synthesis: (1) Split: Input list divided into batches whose combined character count fits within `contextSize Γ— maxCharsMultiplier`. (2) Reduce each batch: Each batch sent to LLM via domain-specific prompt β†’ one summary string per batch. If only one batch exists, returned directly; otherwise summaries concatenated and fed back into step 1 recursively until a single result remains. (3) Graceful truncation: Items exceeding `maxChars` are truncated with `\"...\\n...[truncated]\"`. If every item exceeds the per-item budget (`allowedPerItem`), all items uniformly truncated to avoid infinite loops. (4) Context cancellation: Checked at entry points and before each recursive reduction round; cancellation errors propagate as-is.\n\n**chunkTextWithOverlap**: Text-splitting utility with overlap between adjacent chunks. Appears available for reuse elsewhere but not referenced by any function here.\n\n### Synthesis Core\n\nAll types and functions unexported (`pipelineContext`, `synthesizeNode`). Package-private.\n\n**Recursive Directory Synthesis**: A directory's summary is derived by recursively synthesizing children (subdirectories) and files together. Final result stored in cache under `cache.Modules[node.Path]` and written to disk at `docs/modules/\u003csafe-filename\u003e`.\n\n**Caching Strategy with Hash-Based Invalidation**: Module-level: Stores synthesized summary per directory path. Reused when affected status indicates no changes (short-circuit return). File-level: Stores SHA256 hash + extracted facts in `cache.Files[f]`. Cache hit determined by comparing precomputed or read file hash against cached entry; only re-reads on miss.\n\n**Multi-Step Extraction Pipeline Per File Chunk**: (1) System prompt augmented with current extraction step's prompt. (2) Single LLM call extracts facts from chunk β†’ one fact per step. (3) All steps produce respective facts for the same file β†’ consolidated via `reduceFileFacts`.\n\n**Synthesis Order \u0026 Assembly**: (1) File facts (sorted child name order). (2) Child subdirectory summaries (after all files processed). (3) Combined list reduced to final summary via `reduceInChunks` at directory level.\n\n**Affected-Dir Pruning**: If a directory path has no affected status AND a cached module summary exists β†’ short-circuit return cached value without reprocessing children or files.\n\n### Tree \u0026 Change Detection\n\nExported types: **`FileChange`** β€” `Path string`, `Status string` (\"Added\", \"Modified\", \"Deleted\"). **`DirNode`** β€” `Path string`, `Files []string`, `Children map[string]*DirNode`.\n\n**Three-Phase Algorithm**: (1) Build tree: Given raw file paths, split each on `/`, create intermediate `DirNode` entries for subdirectories, attach files to leaf nodes via `buildTree`. (2) Determine affected directories from changes: For each input change, mark directly changed file as affected (if exists in current set). When a file is Deleted, mark parent directory as affected. Walk recursively and check additional conditions: if node path matches `docs/modules/\u003csafe-name\u003e` but no such doc exists on disk β†’ affected; if node has empty cached metadata entry β†’ affected. (3) Propagate status upward: Recursive walk of `DirNode` tree propagates any `affected` flag from children to parents so a parent is considered affected if any child is.\n\n### JSON Parser Utility\n\n**stripOuterMarkdownFence(input) string**: Single-pass parser that strips markdown or JSON code fences from strings: (1) Trims whitespace from input. (2) Attempts regex match against pattern capturing triple backticks with 3+ chars, optionally followed by `markdown` or `json`, then all content between opening and closing fences (group 1). (3) If matched β†’ returns trimmed captured inner content. (4) If no match β†’ returns original trimmed input unchanged. No error return parameter. `regexp.MustCompile` panics on invalid pattern at package initβ€”unhandled within this file, propagates to any caller importing the package.\n\n### Utility Functions for Filename Generation and Event Logging\n\nThis file provides two unexported utility functions supporting the engine's documentation generation pipeline and optional event logging instrumentation. Neither function communicates with external systems, modifies shared state, or returns errors. All operations are pure in-memory transformations.\n\n**toSafeMarkdownFilename β€” Module Path to Markdown Filename Conversion**: Transforms a module path into a safe markdown documentation filename, enforcing the engine's implicit one-to-one mapping contract between modules and their `.md` documentation files. The root module is special-cased to default to `root.md`. Data flow: (1) Input: a raw string representing a module path (e.g., `\"internal/engine\"`). (2) Character substitution: every `/` character in the input is replaced with `_`. (3) Extension appended: `.md` is concatenated to produce the final filename. (4) Output: a single `string` containing the resulting markdown filename. No filesystem, network, database, or other external I/O occurs during execution. The function returns only a result string; no error type is returned. Invalid input characters (anything other than `/`) are passed through unmodified β€” callers receive no escape hatch for detecting malformed paths. Pure: no mutable state, no package-level globals modified, no struct fields touched.\n\n**makeLogEvent β€” Optional Event Callback Adapter**: Wires an optional callback into a logging pipeline. When invoked with an event type and message pair, it either forwards the call to the registered handler or silently discards it. This makes the logging subsystem purely additive β€” no required consumer exists for the engine's log events; instrumentation is opt-in only. Data flow: (1) Input: a user-provided `onEvent` callback of type `LogEventFunc`. (2) Closure construction: returns a new function accepting `(EventType, message)` pairs. (3) Invocation behavior: if `onEvent == nil`, the returned closure does nothing when called β€” no panic, no error, no log output. If `onEvent` is non-nil, the call is forwarded to it with the provided type and message arguments. No I/O anywhere in this flow. Pure: no mutable state, no package-level globals modified. Silent failure path: when `onEvent == nil`, events are swallowed without notice or error wrapping. The returned function is always valid (non-nil), so callers can invoke it safely regardless of whether a handler was registered.\n\n**Referenced Types**: The following types are referenced in this file but not defined within it: `EventType` β€” categorizes log events into discrete types; carried as the first argument to event handlers and consumed by the returned closure from `makeLogEvent`. `LogEventFunc` β€” function signature for user-provided event handlers. Expected to accept `(EventType, message string)` or equivalent parameters. `Event` (struct) β€” domain model with at least two fields: `Type` (`EventType`) and `Message` (`string`). Events are tracked as discrete, typed records carrying descriptive payloads.\n\n**Concurrency \u0026 State Summary**: Mutable state: None. Both functions operate entirely on their inputs and return values without touching package-level variables or struct fields. Concurrency mechanisms: None. No mutexes, channels, atomic operations, or synchronization primitives are used in this file.\n\n## Security Invariants\n\n### Module Responsibility\n\nThis module enforces two non-negotiable invariants for any code-reducer session: (1) the working directory must remain strictly within the repository root boundary, and (2) only one process may hold an exclusive lock on the shared resource at any given moment. It also maintains a `.gitignore` entry that excludes the lockfile from version control while ensuring it is automatically tracked if absent.\n\nThe module communicates with the outside world indirectly: `security.go` performs filesystem I/O and returns sentinel error values to callers; those sentinels are defined in `errors.go`. No panic recovery, no error wrapping beyond `%w`, no silent swallowing of non-trivial failures occurs within these files.\n\n### Sentinel Error Contract\n\nPackage-level variables: **`ErrPathTraversal`** (`error`) β€” returned when a resolved path escapes outside the repository root boundary. **`ErrLockHeld`** (`error`) β€” returned when another code-reducer process holds an exclusive file lock on the shared resource. Both variables are initialized once via `errors.New` and remain read-only throughout their scope. No mutable state, no concurrency primitives, no panic handling exists in this file. Callers detect violations by matching against these sentinels using `errors.Is(err, ErrPathTraversal)` or equivalent patterns; because the errors are unwrapped as-is (no wrapping), identity is preserved directly through the error chain.\n\n**Domain Concepts**: Path Traversal Violation (`ErrPathTraversal`) β€” signals when a resolved path escapes outside the repository root boundary. This is a classic path traversal / directory escape guardrail. Lock Conflict (`ErrLockHeld`) β€” signals when another process already holds an exclusive file lock, indicating contention on a shared resource.\n\n### Lock Acquisition and Release\n\n**Struct Definition**: `SimpleLock` carries: `lockPath string` (absolute path to the lockfile on disk), `file *os.File` (open file descriptor for the lockfile, or nil if not acquired), `mu sync.Mutex` (protects internal struct fields during `Unlock()`), `closed bool` (marks whether the lock has been released and the file descriptor closed).\n\n**`*SimpleLock.Unlock()`**: Closes the lock file descriptor and removes the lockfile in a thread-safe manner. **Idempotency contract**: Calling `Unlock()` on an already-closed lock returns `nil`. This is intentional, not accidental. Error handling inside the method: If `l.closed == true` β†’ return `nil`. Close the file descriptor (`os.File.Close`) β€” error captured in local variable; if close fails, set `file = nil` so a subsequent close becomes a no-op. Remove the lockfile via `os.Remove`. If both close and remove fail: remove error overwrites prior close error only when close returned without error. If close failed first, any remove error is ignored. This means the final returned error reflects whichever operation last produced an error, not necessarily the most severe one β€” an intentional design choice worth flagging for callers. If `os.IsNotExist(err)` on remove β†’ silently accepted as normal idempotent cleanup.\n\n### Path Resolution\n\n**Signature**: `func SafeResolve(repoRoot, inputPath string) (string, error)`. **Responsibility**: Resolve any input path strictly inside the repository root. The function performs a multi-step containment check to defeat symlink-based escape vectors and physical directory traversal.\n\n**Algorithmic Steps**: (1) Call `filepath.Abs` on `inputPath`. (2) Evaluate symlinks on the existing ancestor via `EvalSymlinks` β€” prevents symlink-based escape from parent directories that exist but are symbolic links. (3) Walk up from the target until a physically-existing directory is found, then re-evaluate symlinks on that ancestor via `EvalSymlinks` again. (4) Rejoin with the remaining suffix components after the physical ancestor boundary. (5) Verify final result is under `resolvedRoot` via relative path check (`filepath.Rel`).\n\n**Error Handling**: Any error from `Abs`, first `EvalSymlinks`, second `EvalSymlinks`, or `Lstat` (non-exist) wraps with descriptive message using `fmt.Errorf(..., err)` β€” underlying cause recovered via `%w`. Unrecognized error from `Lstat` (`!os.IsNotExist`) returns immediately as wrapped error. Path escapes repo root (`..` prefix or non-nil rel) returns sentinel-style error wrapping `ErrPathTraversal` with the original input path.\n\n### Lock Acquisition\n\n**Signature**: `func AcquireLock(repoRoot string) (*SimpleLock, error)`. **Responsibility**: Establish an exclusive process-level lock on a shared resource. Uses `O_EXCL` flag for atomic creation of the lockfile β€” if the lockfile already exists, acquisition fails immediately without silent success.\n\n**Lockfile Lifecycle**: Resolve path calls `SafeResolve(repoRoot)` to obtain absolute lock path. Create atomically opens file with `O_WRONLY | O_CREATE | O_EXCL` β€” atomic exclusive creation prevents race between existence check and open. Record identity writes PID of the acquiring process to the newly created lockfile. Failure cleanup on failure: closes file, removes lockfile.\n\n**Error Handling**: `SafeResolve` fails propagates as-is to caller. File open fails AND error is `os.IsExist` (lock already held) returns wrapped sentinel error wrapping `ErrLockHeld`, including the lock path β€” caller can detect stale-lock condition specifically via `errors.Is(err, ErrLockHeld)`. File open fails with any other error wraps: `\"failed to acquire lock at %s: \u003ccause\u003e\"`. Write PID fails file closed, lockfile removed (cleanup), wrapped error returned.\n\n### Gitignore Maintenance\n\n**Signature**: `func EnsureGitignoreHasLockfile(repoRoot string) (error)`. **Responsibility**: Maintain the convention that the lockfile is excluded from version control but automatically tracked if missing. Appends the lockfile entry to `.gitignore` only if not already present; uses atomic rename via temp file + sync before close to prevent partial writes.\n\n**Algorithmic Steps**: (1) Resolve `.gitignore` path via `SafeResolve(repoRoot)`. (2) Read existing `.gitignore` content (`os.ReadFile`). (3) Parse lines, check if lockfile entry already exists. (4) If not present: construct new content with the lockfile entry appended (with trailing-newline normalization). (5) Create a temp file in same directory as `.gitignore`. (6) Write to temp file β†’ call `Sync()` on it β†’ close it β†’ `Chmod` it. (7) Rename temp file over original `.gitignore`.\n\n**Error Handling**: `SafeResolve` fails propagates as-is. Read fails AND not `os.IsNotExist` (file exists but read error) wraps: `\"error reading .gitignore: \u003ccause\u003e\"` β€” caller must handle missing-file vs I/O-error distinction separately. CreateTemp fails wraps with descriptive message including `%w`. Write to temp file fails wraps; temp file is NOT cleaned up (defer handles close + remove, but write error returns before cleanup path executes). Sync fails wraps β€” note that `os.Chmod` and subsequent rename may have already run on a partially-written temp file if this error occurs late in the sequence. Close fails wraps. Chmod fails wraps. Rename (atomic swap) fails wraps with descriptive message.\n\nThe deferred function (`tmpFile.Close()` + `os.Remove(tmpName)`) runs regardless of return path. Because the function returns before the defer has a chance to execute in some failure paths, cleanup may not happen for intermediate states β€” but since we're using temp file with unique name and same-directory rename, the OS-level temp file is cleaned by `os.Remove` in the defer on any error return path. If *write* fails mid-stream before close/rename, the deferred Remove will clean up the partially-written temp file.\n\n### Data Flow Summary\n\n```\n[Initialize] β†’ [Ensure path safety for all repo-root operations (SafeResolve)] β†’ \n[If no process holds the lock] β†’ [Create lockfile with PID via O_EXCL (AcquireLock)] β†’ \n[Run business logic within locked session] β†’ \n[Append entry to .gitignore if needed (EnsureGitignoreHasLockfile)]\n```\n\nThe two non-obvious constraints are: (1) path resolution walks symlinked ancestors before rejoining suffix components, and (2) gitignore updates use atomic rename with explicit sync before close.\n\n**Key Patterns Observed Across the Module**: Error wrapping throughout β€” All functions use `fmt.Errorf(..., err)` with `%w`, making root-cause recovery possible via `errors.Is()`. Custom sentinel errors β€” At least two custom error variables referenced: `ErrPathTraversal` and `ErrLockHeld`. These enable typed error detection by callers without requiring unwrapping logic on this file's side. No panic anywhere β€” All failure paths return to caller with an error value. Idempotent operations deliberately silent β€” `Unlock()` on already-closed lock returns nil; missing `.gitignore` in EnsureGitignoreHasLockfile returns nil. These are design choices, not accidentals. Atomic rename pattern for .gitignore β€” Write to temp β†’ Sync β†’ Close β†’ Chmod β†’ Rename is the standard atomic-update pattern. The `Sync()` call before close is explicit and unusual (normally deferred), but here it's called eagerly before close in this implementation. No global package-level mutable state exists in either file β€” all state is either local to a function or encapsulated within the `SimpleLock` receiver, which is accessed only through its methods.\n\n## Tools Subsystem\n\n### Module Responsibility\n\nThe `internal/tools` package provides repository-scanning primitives for a codebase-analysis tool. It exposes two functional domains: **safe local file I/O** (`file_tools.go`) and **git subprocess orchestration** (`git_tools.go`). Together they enable the caller to discover source-code files, classify their content type, read/write artifacts safely, and validate that the target directory is a git work-tree before proceeding.\n\nData flow follows a two-phase pattern: (1) `VerifyGitRepo` (or an equivalent pre-flight check) confirms the working directory corresponds to a valid git repository by invoking `git rev-parse --is-inside-work-tree`. (2) Once validated, `DiscoverCodeFiles` walks the entire tree under that root, applying layered ignore rules and returning only text-source files. Both domains share these architectural constraints: **no panics**, **error wrapping** (stderr is preserved for diagnostics), and **no network/DB calls**. All I/O operates within a single local process or on the local filesystem.\n\n### File Operations\n\n**Safe Read / Write Primitive**: `ReadFileSafely` and `WriteFileSafely` implement TOCTOU-safe file operations using the temp-file + rename pattern. Shared sequence: (1) Resolve a virtual path to an absolute filesystem path via `SafeResolve`. (2) On write: create parent directories with `defaultDirPerm`; on read: no directory creation. (3) Open or create a temporary file in the target directory using `os.CreateTemp`. (4) Write content (for writes) or read from the resolved path (for reads). (5) Close and sync (writes only); close (reads only). (6) Rename temp into final destination for writes; return contents for reads.\n\n**Symlink detection**: Both paths verify the target is not a symlink via `os.Lstat`/`f.Stat`. Pre-open checks prevent TOCTOU swap attacks where an attacker replaces a real file with a symlink between resolution and open. Post-open checks catch any state changes after the initial open.\n\n**Error semantics**: `ReadFileSafely`: returns `nil, err` on failure; wraps each step with context strings (`failed to open file`, `failed to lstat`, etc.). Symlink detection yields an explicit \"security violation\" error rather than a generic I/O error. `WriteFileSafely`: all failure paths return wrapped errors (directory creation, temp write, sync, close, chmod, rename). No panic usage; no sentinel values. Deferred cleanup: Both functions defer `f.Close()` for reads and use deferred `os.Remove` in `WriteFileSafely` to clean up temp files if the rename fails.\n\n**Gitignore Handling**: **`LoadGitignore(repoRoot)`**: Opens `.gitignore` under `repoRoot`. If absent (`os.IsNotExist`), returns `nil, nil` β€” no error raised. Any other open/read failure is returned unwrapped. **`ShouldIgnoreFile(repoRoot, relPath, gitIgnore)`**: Applies layered rules: (1) Check against compiled `GitIgnore` patterns (user-supplied + `.gitignore`). (2) Reject any path component starting with `.` or ending with `.egg-info`, regardless of other rules. (3) If extension matches a known text-source language list, accept as code. (4) For unknown extensions: resolve absolute path and scan first 1024 bytes for null bytes; binaries return `true`.\n\n**Side effect note**: `ShouldIgnoreFile` opens the file on every invocation to classify unknown extensions β€” this is a real I/O side effect per call, not cached.\n\n### Code Discovery Pipeline\n\n**Input**: repository root + list of ignore patterns (strings). **Output**: slice of relative source-code paths.\n\n**Algorithmic flow**: (1) Compile user-supplied ignore patterns and `.gitignore` entries into a single `GitIgnore` object via `LoadGitignore`. (2) Walk the entire tree recursively using `filepath.WalkDir`. For each entry: Directory: if name starts with `.` or matches any compiled pattern, skip subtree entirely. File: delegate to `ShouldIgnoreFile`. (3) Collect passing files into result slice.\n\n**Walk error handling**: If a walk callback returns an error for a single entry, the error is printed to `stderr` and that item is skipped (`return nil`). The overall `err` from `WalkDir` propagates at the end β€” terminal fatal errors are returned; non-fatal entries are swallowed silently.\n\n### Binary Classification\n\n**IsBinaryFile**: Scans first 1024 bytes for null bytes to distinguish text source files from binary content. Returns `true` (ignored) on any I/O error other than EOF β€” this \"fail-closed\" assumption treats unreadable files as likely binary rather than propagating the actual error.\n\n**Constants**: `binaryDetectionBufSize` is lowercase and package-private; not exported.\n\n### Git Tools\n\n**External Command Execution (`RunGit`)**: Spawns `git --no-pager` via `exec.Command` with any additional arguments from the caller. The subprocess runs within `repoRoot` as its working directory. Stdout/stderr are captured into in-memory `bytes.Buffer`; no file, network, or DB I/O occurs.\n\n**Failure handling**: On error, wraps the original with stderr content:\n```\nfmt.Errorf(\"git command failed: %v, stderr: %s\", err, trimmedErr)\n```\nReturns `(string, error)` β€” partial stdout is still delivered alongside the error. No panics.\n\n**Repository Validation (`VerifyGitRepo`)**: Delegates to `RunGit` with subcommand `rev-parse --is-inside-work-tree`. If this invocation results in an error, returns a fixed message: `\"not a git repository (or any of the parent directories)\"`. The original stderr is discarded at this wrapper layer β€” normalized to a single sentinel-style message for downstream callers.\n\n### Concurrency \u0026 State Analysis\n\n**Mutable state**: None detected across both files. All constants are `const` (compile-time); all variables (`safePath`, `dir`, `tmpFile`, `tmpName`, `patterns`, `files`, `buf`, `cmd`, `stdout`, `stderr`, etc.) are function-local and never escape their defining scope. No package-level global state is modified.\n\n**Concurrency mechanisms**: None detected. No `sync.Mutex`, channels, `atomic` types, or other synchronization primitives. Shared mutable state does not exist to protect.\n\n### Error Handling Summary\n\n| Pattern | Where |\n|---|---|\n| Wrap-and-return (preserve stderr) | `RunGit` |\n| Normalize to sentinel message | `VerifyGitRepo` |\n| Sentinel + wrap with context | `ReadFileSafely`, `WriteFileSafely`, `LoadGitignore` |\n| Boolean return, resolve failures as \"ignored\" | `ShouldIgnoreFile` |\n| Swallow per-entry walk errors, propagate terminal | `DiscoverCodeFiles` |\n| Fail-closed (I/O error β†’ binary assumption) | `IsBinaryFile` |\n\n**No panics** observed across either file. All errors are returned to callers as Go values.", + "cmd": "# Module: `cmd` β€” CLI Command Orchestration and Entry Point\n\n## Responsibility\n\nThe `cmd` module defines the root cobra command (`RootCmd`) for **Code-Reducer**, a documentation agent that writes and maintains project wikis. It registers subcommands (`init`, `update`, `setup`), resolves configuration from merged sources (flags + file), routes engine events to stdout/stderr, handles OS signals for graceful shutdown, and delegates all substantive work to the unshown `executeCommand` entry point.\n\n## Data Flow\n\n```\nUser invokes code-reducer CLI\n β”‚\n β”œβ”€β”€ 1. Register subcommands via init() / update.go (package-level)\n β”‚\n β”œβ”€β”€ 2. Parse persistent flags (--model-id, --num-ctx) on RootCmd\n β”‚\n β”œβ”€β”€ 3. Resolve current working directory; fail if not a git repo\n β”‚\n β”œβ”€β”€ 4. Check configuration state\n β”‚ β”œβ”€β”€ No config + TTY β†’ RunSetupFlow(repoRoot)\n β”‚ └── Config exists β†’ Continue\n β”‚\n β”œβ”€β”€ 5. Merge configuration sources: flags + file-based config (config.ResolveConfig)\n β”‚\n β”œβ”€β”€ 6. Determine operation mode (init / update / run) and validate state\n β”‚ β”œβ”€β”€ init β†’ verify not already initialized\n β”‚ β”œβ”€β”€ update β†’ verify project initialized\n β”‚ └── other modes β†’ proceed to engine\n β”‚\n β”œβ”€β”€ 7. Install signal handler for INT/TERM via signal.NotifyContext\n β”‚\n β”œβ”€β”€ 8. Instantiate documentation engine and execute (engine.NewRunner.Run)\n β”‚ └── Engine emits typed events during processing\n β”‚ β”œβ”€β”€ Status β†’ fmt.Println(ev.Message) [stdout]\n β”‚ └── Error β†’ fmt.Fprintf(os.Stderr, \"Error: %s\\n\", ev.Message) [stderr]\n β”‚\n └── 9. Return result (success or wrapped error via fmt.Errorf(\"…: %w\", err))\n```\n\n## Subcommand Registration\n\n### `init` β€” Documentation Initialization\n\nThe `cmd init` subcommand is defined and registered during package initialization in `init.go`. It delegates all processing to the unshown `executeCommand(\"init\")`, which performs repository scanning and generates an initial set of wiki markdown pages. The command itself does not perform any I/O, error wrapping, or logic beyond registration and delegation.\n\n### `update` β€” Incremental Wiki Update\n\nThe `cmd update` subcommand is defined in `update.go`. On invocation, it calls `executeCommand(\"update\")`, which scans the repository for files changed since the last documented commit and regenerates corresponding wiki pages. The actual scanning and regeneration logic resides outside this file; `update.go` only defines the entry point.\n\n### `setup` β€” Interactive Configuration Setup\n\nThe `cmd setup` subcommand is exported as `RunSetupFlow(repoRoot string) error`. It provides an interactive CLI flow that guides users through configuring all application settings, persisting them to `.code-reducer.yaml`. The configuration captures nine fields: model ID, base URL, context size, ignore patterns, docs directory path, and four distinct system prompts (extraction steps, module synthesis, architecture description, file-fact consolidation).\n\nThe setup flow operates as follows:\n1. Attempt to load existing config via `config.LoadConfig(repoRoot)`. Errors are silently swallowed; defaults apply if loading fails.\n2. Prompt user sequentially for each setting. Existing values appear in brackets as hints inside the prompt. Empty input or `clear`/`none` retains the existing/fallback value. Ignore patterns accept comma-separated values and replace the entire list on any non-empty input.\n3. Assemble final configuration by merging user inputs with preserved existing values (prompts update only if new content is provided; ignores fully override).\n4. Persist via `config.SaveConfig(repoRoot, newCfg)`. Success confirms the file path; failure returns a wrapped error `\"error saving configuration: %w\"`.\n\n## Configuration Resolution and Validation\n\n### State Initialization\n\nPackage-level variables are declared and assigned exactly once at initialization:\n- `modelIDFlag` (`string`) β€” set via flag parsing on RootCmd. Only read subsequently.\n- `numCtxFlag` (`string`) β€” same as above; never reassigned after declaration.\n- `RootCmd` (`*cobra.Command`) β€” initialized with a literal; no subsequent writes occur in this file.\n\n### Configuration Lifecycle Modes\n\nThe module enforces two distinct modes for project setup:\n- **Init mode**: Creates/initializes documentation infrastructure (wiki structure). Fails if already initialized, requiring `update` to refresh later.\n- **Update mode**: Refreshes existing documentation. Fails if the project has not been initialized yet.\n\n### Configuration Resolution Priority\n\nThe merged configuration is resolved from three sources:\n1. Persistent command-line flags (`--model-id`, `--num-ctx`)\n2. File-based config (from project directory)\n3. These are merged together into a single resolved config object via `config.ResolveConfig(repoRoot, modelIDFlag, numCtxFlag)`\n\n## Signal Handling and Graceful Shutdown\n\nA signal handler captures `INT` (`os.Interrupt`) and `TERM` (`syscall.SIGTERM`) signals via `signal.NotifyContext`. A deferred call to `stop()` ensures the context is cancelled when a signal arrives. The engine runner's returned error is then wrapped as `\"documentation run failed: …\"`.\n\n## Error Handling Patterns\n\n### Wrap Pattern (100% of observed paths)\n\nEvery error originates from a sub-function call, is immediately wrapped with `fmt.Errorf(\"…: %w\", err)`, and propagated to the next level or returned to cobra's main handler. Examples include:\n- `os.Getwd()` failure β†’ wrapped as `\"failed to get current working directory: …\"`\n- `tools.VerifyGitRepo` error β†’ passed through unchanged (already a wrapped/typed error)\n- `RunSetupFlow` error β†’ passed through unchanged\n- `config.ResolveConfig` error β†’ passed through unchanged\n- `engine.NewRunner(cfg).Run(...)` result β†’ wrapped as `\"documentation run failed: …\"`\n\n### Sentinel / Early-Return Logic\n\nThe function returns immediately if the working directory cannot be obtained (`os.Getwd()` fails). It returns early with a typed error from `tools.VerifyGitRepo` without wrapping. If stdin is not a terminal and config is missing, it returns an explicit sentinel message: `\"configuration file … does not exist… Please run 'code-reducer setup'…\"`. \"Init already done\" β†’ returns wrapped error telling user to use `update`. \"Not initialized yet\" (during update) β†’ returns wrapped error telling user to run `init` first.\n\n### Nil Error Path\n\nIf `executeCommand` returns `nil`, the command succeeds silently. No logging or confirmation output is emitted in this file.\n\n## Concurrency Characteristics\n\nNo mutexes, channels, goroutines, atomics, or other concurrency primitives are used in any of these files. All operations execute sequentially on the main thread. Concurrency relies solely on `context.Context` cancellation via OS signals; no shared-mutable-state synchronization primitives are present.", + "internal": "# Code Reducer β€” Architecture Documentation\n\n## Module Responsibility \u0026 Data Flow\n\nCode Reducer is a multi-phase LLM analysis pipeline that generates technical documentation from source code. The system operates across four subsystems: **config** (pipeline configuration and lifecycle), **engine** (recursive, context-window-aware synthesis orchestration), **security** (path containment and process mutual exclusion), and **tools** (repository scanning and safe I/O primitives).\n\nData flow is sequential across subsystem boundaries: `internal/config` resolves a fully-populated `*Config` via multi-source priority merging; `internal/engine` consumes that config to drive an LLM pipeline; `internal/security` enforces path containment for all filesystem operations the engine performs; `internal/tools` provides file I/O and git discovery primitives used by both security and engine subsystems.\n\nNo shared mutable state exists across subsystem boundaries. Each module operates on caller-provided arguments and returns results. No mutexes, channels, or atomic operations appear anywhere in the codebase.\n\n---\n\n## Subsystem: internal/config\n\n### Configuration Types and Defaults\n\n`config.go` declares three exported types: `ExtractionStep` (per-phase identifiers carrying a name string and prompt text), `Config` (aggregate configuration container referenced by all exported functions), and compile-time constants for environment variable keys, Ollama default values (`OllamaDefaultBaseURL = \"http://localhost:11434\"`, `OllamaDefaultModelID = \"ornith:9b\"`, `OllamaDefaultNumCtx = 8192`), file-system defaults (`DefaultDocsDir = \"wiki\"`, `ConfigFileName = \".code-reducer.yaml\"`), and multiline prompt templates.\n\n`DefaultExtractionSteps` is a pre-populated slice containing four named phases: `API_SIGNATURES`, `BUSINESS_LOGIC`, `STATE_AND_CONCURRENCY`, `ERRORS_AND_SIDE_EFFECTS`. Initialized at declaration only; no post-initialization mutations occur in this file. Internal helper variable `configFilePerm` (value 0600) is referenced by `io.go` but not declared here. No exported interfaces or methods exist on these types.\n\n### Configuration File Lifecycle\n\n`io.go` provides three exported functions: `ConfigExists(cwd string)` returns a boolean, `LoadConfig(cwd string)` returns `(*Config, error)`, and `SaveConfig(cwd string, cfg *Config)` returns an error. All external type references (`Config`) come from the same package.\n\n**Atomic Write Pattern for SaveConfig:** The save operation does not write directly to the target path. The sequence is: marshal `cfg` into YAML bytes β†’ apply formatting normalization (collapsing single blank lines to double blank lines before known section headers: `system_prompt`, `module_synthesis_prompt`, `architecture_prompt`, `file_fact_consolidation_prompt`, `extraction_steps`, `ignore`) β†’ create a temp file matching `.tmp.*` in the same directory β†’ write normalized YAML into it β†’ sync and close β†’ set permissions via `os.Chmod` β†’ atomically rename temp over original. Readers never observe a partially-written config during save because the old file remains intact until the new one is fully ready to replace it.\n\n**Load-Save Roundtrip Integrity:** Loading preserves whatever formatting was originally written; normalization is applied only on write, not read. This decouples the in-memory representation from on-disk formatting expectations. `LoadConfig` returns a fresh `*Config` via unmarshaling with no shared state. Errors from `os.ReadFile` propagate directly; YAML parse failures are wrapped with a `\"failed to parse yaml config\"` prefix chained via `%w`; marshal failures similarly wrap a descriptive message.\n\n**Error Propagation Strategy:** When `os.ReadFile` fails in `LoadConfig`, the OS error is returned unwrapped. Parse and write failures across all save-path operations (temp create, write, sync, close, chmod, rename) are wrapped with context via `%w`. `ConfigExists` uses an `err == nil` check instead of returning the error; no wrapping; raw error is reused as a boolean condition. A deferred function calls `tmpFile.Close()` and `os.Remove(tmpName)` in `SaveConfig`; errors from these cleanup calls are swallowed (not returned). On success, `Close` runs twice on the temp fileβ€”Go allows this safelyβ€”and the defer's `Remove` returns an error that is silently swallowed because the original config now holds the renamed data.\n\n### Multi-Source Configuration Resolution\n\n`resolve.go` exposes a single exported function: `ResolveConfig(repoRoot, modelIDFlag, numCtxFlag string)` returning `(*config.Config, error)`. Internal helper `mergeAndDeduplicate` is lowercase and not part of the public surface. Only usage (not definition) of package-level types (`Config`, `OllamaDefaultModelID`, `CodeReducerModelIDEnvKey`, `DefaultExtractionSteps`) is visible here.\n\n**Resolution Priority System:** Each configurable field follows its own independent priority chain, lowest to highest: **defaults β†’ YAML config file β†’ environment variables β†’ CLI flags**. The merge-and-deduplicate logic operates on local function scopes with no cross-goroutine visibility and no shared mutable state.\n\n**Field-Specific Priority Chains:** `ModelID` applies Default \u003e YAML \u003e Env Var \u003e CLI Flag. `OllamaBaseURL` applies Default \u003e YAML \u003e Env Var. `OllamaNumCtx` (context size) applies Default \u003e YAML \u003e Env Var \u003e CLI Flag. `DocsDir` applies Default \u003e YAML. System prompts (`SystemPrompt`, `ModuleSynthesisPrompt`, `ArchitecturePrompt`, `FileFactConsolidationPrompt`) all start with default, then override with YAML if non-empty.\n\n**Validation Rules:** Numeric fields: CLI flag and env var values are parsed as integers; a value must be greater than zero to take effect. Invalid or non-positive values fall through silently. String fields: empty string is treated as \"not set\" β€” the lower-priority default remains active. Config file absence (`config.yml`): not an error; if missing, an empty `Config{}` struct is created and defaults are applied. Other parse errors are surfaced.\n\n**Algorithmic Flow for ResolveConfig:** (1) Load YAML config β€” if it exists and parses successfully; otherwise start with an empty struct (missing-file case). (2) Resolve extraction steps β€” use YAML value if present, else fall back to `DefaultExtractionSteps`. (3) Deduplicate ignore list from the loaded config using a map-based seen-set while preserving first-seen order. (4) For each field, apply its specific priority chain: start with built-in default constant β†’ override with YAML (if non-empty) β†’ override with environment variable (if set and valid for numeric types) β†’ override with CLI flag (highest priority; numeric types require successful integer parse and positive value). (5) Return the fully resolved `Config` struct, or an error if the YAML file fails to parse (excluding missing-file case).\n\n**Error Handling in ResolveConfig:** When `os.IsNotExist(err)` is true during config loading, defaults apply with no error returned. Any other error (parse failure, permission denied) is wrapped with a prefix and returned as `(*Config, nil, error)` β€” the config pointer is `nil`. Silent branches: string comparisons (`!= \"\"`, `\u003e 0`) short-circuit without errors. `strconv.Atoi` failures are caught inline; env/flag values simply not used. Map lookups for deduplication have no error path.\n\n**Always-On Return:** The function always returns a non-nil tuple `(*Config, error)` β€” either a populated config with `nil` error, or a `nil` config with an error string. No panics anywhere in this file.\n\n---\n\n## Subsystem: internal/engine\n\n### Constants \u0026 Types\n\n`constants.go` declares compile-time constants: `defaultHTTPTimeout` (10 min), `maxErrorBodyBytes` (1024 bytes), `defaultChunkOverlap` (800 chars), `minNumCtxFloor` (512 tokens), `contextWindowAllocRatio` (0.75), `maxCharsMultiplier` (3), `metadataFileName` (`.metadata.json`), `agentsFileName` (`AGENTS.md`), `defaultDirPerm` (0755). Type alias `LogEventFunc` carries signature `func(EventType, string)` with no business rules defined here.\n\n### LLM Client Communication\n\nUnexported except `Message` struct (`Role string`, `Content string`). The package-private `llmClient` carries: model ID, base URL, context count, HTTP client pointer. All set once during construction; never modified afterward within this file. No mutexes or channels used.\n\n**Algorithmic Flow:** (1) Prepare payload: assemble request body by prepending `systemPrompt` as a system-role message before any user-supplied messages, serialize as JSON to Ollama `/api/chat`. (2) Send HTTP request: single synchronous POST with fixed timeout; no retries (fail-fast policy). (3) Handle response: on 200 OK β†’ deserialize body β†’ extract `message.content` field. Non-2xx status code β†’ error string containing both HTTP status and raw response payload. (4) Return result: extracted text content or wrapped error describing what went wrong.\n\n**Error Handling Patterns:** Marshal failure wraps via `%w` for caller inspection via `errors.Is`. HTTP request construction returns unwrapped; context lost for callers. Network/transport error returns unwrapped; timeout vs DNS vs connect indistinguishable to caller. Success-path body read returns unwrapped (rare). JSON unmarshal failure wraps via `%w`. Non-OK status response uses custom message discarding read error (`_`); uses undefined `maxErrorBodyBytes` constant. No retry logic.\n\n### Runner \u0026 Entry Point\n\n`Runner` wraps a config pointer and exposes a single `Run(ctx, repoRoot, mode, onEvent)` method that is the only public entry point into the pipeline. It performs four sequential operations: (1) Gitignore lockfile maintenance β€” attempts to append the lockfile path to `.gitignore`. Failure is logged via `onEvent(EventStatus, ...)` and execution continues; no error returned. (2) Repository lock acquisition β€” calls `security.AcquireLock(repoRoot)`. On failure, returns wrapped error: `\"failed to acquire repository lock: %w\"`. No partial work proceeds without the lock. (3) Mode dispatch β€” constructs an LLM client from config fields (`cfg.ModelID`, `cfg.OllamaBaseURL`, `cfg.OllamaNumCtx`) and instantiates an `orchestrator`. Dispatches to either `RunInit` or `RunUpdate`; any other mode returns `\"unsupported mode: %s\"`. (4) Deferred lock release β€” `defer lock.Unlock()` runs after the pipeline completes, regardless of success or failure.\n\nThe `Runner.cfg` field is set once in `NewRunner(cfg)` and never mutated within this file. The pointer type allows external reassignment but no such mutation occurs here.\n\n### Orchestrator \u0026 Pipeline Modes\n\nUnexported struct with two exported constants on the `EventType` type alias: `EventStatus = \"status\"` and `EventError = \"error\"`. The unexported `orchestrator` struct carries a method receiver; its public surface is empty.\n\n**RunInit β€” Cold Start Flow:** (1) setupPipeline() β†’ discover files, compute hashes, load .gitignore (swallowed), load cache (swallowed). (2) Mark every directory as affected (full regeneration). (3) Synthesize root summary from full tree via LLM calls. (4) Generate architecture.md + quickstart.md using root summary. (5) Write agent guidelines file (append if marker missing, overwrite otherwise). (6) Persist cache to disk β†’ complete.\n\n**RunUpdate β€” Incremental Refresh Flow:** (1) setupPipeline() β†’ discover current files, compute hashes. (2) If extraction steps changed β†’ invalidate entire cache (.StepsHash mismatch resets Files and Modules maps). (3) Classify all discovered files into Added/Modified/Deleted buckets by comparing current vs cached SHA-256 hashes. (4) Build directory tree from allowed (existing + new) files only. (5) Purge stale module summaries: for any `cache.Modules` path not represented in the live tree, delete both the cache entry and the physical `.md` file (`_ = os.Remove(...)` β€” error swallowed). (6) Compute affected directories; propagate upward through parent dirs. (7) If `len(affectedDirs) == 0` β†’ return early (no-op, docs are up to date). (8) Synthesize root summary from the affected region via LLM calls. (9) Re-generate architecture.md + quickstart.md only if `.` is in `affectedDirs`, or either file does not exist on disk, or resolution fails (treated as \"not exists\"). (10) Persist cache β†’ complete.\n\n**Pipeline Context:** Unexported struct carrying: LLM client pointer, repo root string, config pointer, `*MetadataCache` pointer, affected directory set map, precomputed file hashes map, and event logger callback. Constructed fresh per invocation; no shared instances exist across goroutines in this file.\n\n### Cache Persistence Layer\n\nTwo structs with no methods: **`FileCacheEntry`** β€” pairs a SHA256 digest string (`sha256`) with extracted facts string (`facts`). Stored in `MetadataCache.Files map[string]FileCacheEntry`. **`MetadataCache`** β€” carries version int, steps hash string, files map, and modules map (module dir β†’ synthesized summary).\n\n**IsInitialized(repoRoot, docsDir) bool:** Returns true only if the cache file exists AND is readable in one call. Reuses the existence check from `loadMetadataCache`. No error return.\n\n**saveMetadataCache(repoRoot, docsDir, cache) error:** Serializes to `{docsDir}/{metadataFileName}` with indented JSON (`json.MarshalIndent(cache, \"\", \" \")`). Marshaling errors propagate directly; write errors pass through `tools.WriteFileSafely`. The version field is pinned to the current value on every save.\n\n**Version Compatibility Guard:** On load: if `cache.Version != currentCacheVersion` (currently 1), returns a fresh zero-valued `MetadataCache{}` with nil error. No warning, no panicβ€”caller sees success with an empty cache. This prevents stale metadata from corrupting future runs without explicit migration handling.\n\n**Fault-Tolerant Initialization:** If the file does not exist (`os.ErrNotExist` sentinel matched via `errors.Is`), returns a populated zero-valued cache with nil error. If it exists but cannot be read or unmarshaled, returns wrapped error `\"failed to read metadata cache: %w\"`. Callers can distinguish \"not yet initialized\" from \"corrupt data.\"\n\n### Chunking \u0026 Reduction Engine\n\nAll unexported; package-private. Implements a **context-aware recursive reduction** for LLM-based synthesis: (1) Split: Input list divided into batches whose combined character count fits within `contextSize Γ— maxCharsMultiplier`. (2) Reduce each batch: Each batch sent to LLM via domain-specific prompt β†’ one summary string per batch. If only one batch exists, returned directly; otherwise summaries concatenated and fed back into step 1 recursively until a single result remains. (3) Graceful truncation: Items exceeding `maxChars` are truncated with `\"...\\n...[truncated]\"`. If every item exceeds the per-item budget (`allowedPerItem`), all items uniformly truncated to avoid infinite loops. (4) Context cancellation: Checked at entry points and before each recursive reduction round; cancellation errors propagate as-is.\n\n**chunkTextWithOverlap:** Text-splitting utility with overlap between adjacent chunks. Appears available for reuse elsewhere but not referenced by any function here.\n\n### Synthesis Core\n\nAll types and functions unexported (`pipelineContext`, `synthesizeNode`). Package-private.\n\n**Recursive Directory Synthesis:** A directory's summary is derived by recursively synthesizing children (subdirectories) and files together. Final result stored in cache under `cache.Modules[node.Path]` and written to disk at `docs/modules/\u003csafe-filename\u003e`.\n\n**Caching Strategy with Hash-Based Invalidation:** Module-level: Stores synthesized summary per directory path. Reused when affected status indicates no changes (short-circuit return). File-level: Stores SHA256 hash + extracted facts in `cache.Files[f]`. Cache hit determined by comparing precomputed or read file hash against cached entry; only re-reads on miss.\n\n**Multi-Step Extraction Pipeline Per File Chunk:** (1) System prompt augmented with current extraction step's prompt. (2) Single LLM call extracts facts from chunk β†’ one fact per step. (3) All steps produce respective facts for the same file β†’ consolidated via `reduceFileFacts`.\n\n**Synthesis Order \u0026 Assembly:** (1) File facts (sorted child name order). (2) Child subdirectory summaries (after all files processed). (3) Combined list reduced to final summary via `reduceInChunks` at directory level.\n\n**Affected-Dir Pruning:** If a directory path has no affected status AND a cached module summary exists β†’ short-circuit return cached value without reprocessing children or files.\n\n### Tree \u0026 Change Detection\n\nExported types: **`FileChange`** β€” `Path string`, `Status string` (\"Added\", \"Modified\", \"Deleted\"). **`DirNode`** β€” `Path string`, `Files []string`, `Children map[string]*DirNode`.\n\n**Three-Phase Algorithm:** (1) Build tree: Given raw file paths, split each on `/`, create intermediate `DirNode` entries for subdirectories, attach files to leaf nodes via `buildTree`. (2) Determine affected directories from changes: For each input change, mark directly changed file as affected (if exists in current set). When a file is Deleted, mark parent directory as affected. Walk recursively and check additional conditions: if node path matches `docs/modules/\u003csafe-name\u003e` but no such doc exists on disk β†’ affected; if node has empty cached metadata entry β†’ affected. (3) Propagate status upward: Recursive walk of `DirNode` tree propagates any `affected` flag from children to parents so a parent is considered affected if any child is.\n\n### JSON Parser Utility\n\n**stripOuterMarkdownFence(input) string:** Single-pass parser that strips markdown or JSON code fences from strings: (1) Trims whitespace from input. (2) Attempts regex match against pattern capturing triple backticks with 3+ chars, optionally followed by `markdown` or `json`, then all content between opening and closing fences (group 1). (3) If matched β†’ returns trimmed captured inner content. (4) If no match β†’ returns original trimmed input unchanged. No error return parameter. `regexp.MustCompile` panics on invalid pattern at package initβ€”unhandled within this file, propagates to any caller importing the package.\n\n### Utility Functions for Filename Generation and Event Logging\n\nThis file provides two unexported utility functions supporting the engine's documentation generation pipeline and optional event logging instrumentation. Neither function communicates with external systems, modifies shared state, or returns errors. All operations are pure in-memory transformations.\n\n**toSafeMarkdownFilename β€” Module Path to Markdown Filename Conversion:** Transforms a module path into a safe markdown documentation filename, enforcing the engine's implicit one-to-one mapping contract between modules and their `.md` documentation files. The root module is special-cased to default to `root.md`. Data flow: (1) Input: a raw string representing a module path (e.g., `\"internal/engine\"`). (2) Character substitution: every `/` character in the input is replaced with `_`. (3) Extension appended: `.md` is concatenated to produce the final filename. (4) Output: a single `string` containing the resulting markdown filename. No filesystem, network, database, or other external I/O occurs during execution. The function returns only a result string; no error type is returned. Invalid input characters (anything other than `/`) are passed through unmodified β€” callers receive no escape hatch for detecting malformed paths. Pure: no mutable state, no package-level globals modified, no struct fields touched.\n\n**makeLogEvent β€” Optional Event Callback Adapter:** Wires an optional callback into a logging pipeline. When invoked with an event type and message pair, it either forwards the call to the registered handler or silently discards it. This makes the logging subsystem purely additive β€” no required consumer exists for the engine's log events; instrumentation is opt-in only. Data flow: (1) Input: a user-provided `onEvent` callback of type `LogEventFunc`. (2) Closure construction: returns a new function accepting `(EventType, message)` pairs. (3) Invocation behavior: if `onEvent == nil`, the returned closure does nothing when called β€” no panic, no error, no log output. If `onEvent` is non-nil, the call is forwarded to it with the provided type and message arguments. No I/O anywhere in this flow. Pure: no mutable state, no package-level globals modified. Silent failure path: when `onEvent == nil`, events are swallowed without notice or error wrapping. The returned function is always valid (non-nil), so callers can invoke it safely regardless of whether a handler was registered.\n\n**Referenced Types:** The following types are referenced in this file but not defined within it: `EventType` β€” categorizes log events into discrete types; carried as the first argument to event handlers and consumed by the returned closure from `makeLogEvent`. `LogEventFunc` β€” function signature for user-provided event handlers. Expected to accept `(EventType, message string)` or equivalent parameters. `Event` (struct) β€” domain model with at least two fields: `Type` (`EventType`) and `Message` (`string`). Events are tracked as discrete, typed records carrying descriptive payloads.\n\n**Concurrency \u0026 State Summary:** Mutable state: None. Both functions operate entirely on their inputs and return values without touching package-level variables or struct fields. Concurrency mechanisms: None. No mutexes, channels, atomic operations, or synchronization primitives are used in this file.\n\n---\n\n## Subsystem: internal/security\n\n### Module Responsibility\n\nThis module enforces two non-negotiable invariants for any code-reducer session: (1) the working directory must remain strictly within the repository root boundary, and (2) only one process may hold an exclusive lock on the shared resource at any given moment. It also maintains a `.gitignore` entry that excludes the lockfile from version control while ensuring it is automatically tracked if absent.\n\nThe module communicates with the outside world indirectly: `security.go` performs filesystem I/O and returns sentinel error values to callers; those sentinels are defined in `errors.go`. No panic recovery, no error wrapping beyond `%w`, no silent swallowing of non-trivial failures occurs within these files.\n\n### Sentinel Error Contract\n\nPackage-level variables: **`ErrPathTraversal`** (`error`) β€” returned when a resolved path escapes outside the repository root boundary. **`ErrLockHeld`** (`error`) β€” returned when another code-reducer process holds an exclusive file lock on the shared resource. Both variables are initialized once via `errors.New` and remain read-only throughout their scope. No mutable state, no concurrency primitives, no panic handling exists in this file. Callers detect violations by matching against these sentinels using `errors.Is(err, ErrPathTraversal)` or equivalent patterns; because the errors are unwrapped as-is (no wrapping), identity is preserved directly through the error chain.\n\n**Domain Concepts:** Path Traversal Violation (`ErrPathTraversal`) β€” signals when a resolved path escapes outside the repository root boundary. This is a classic path traversal / directory escape guardrail. Lock Conflict (`ErrLockHeld`) β€” signals when another process already holds an exclusive file lock, indicating contention on a shared resource.\n\n### Lock Acquisition and Release\n\n**Struct Definition:** `SimpleLock` carries: `lockPath string` (absolute path to the lockfile on disk), `file *os.File` (open file descriptor for the lockfile, or nil if not acquired), `mu sync.Mutex` (protects internal struct fields during `Unlock()`), `closed bool` (marks whether the lock has been released and the file descriptor closed).\n\n**`*SimpleLock.Unlock()`:** Closes the lock file descriptor and removes the lockfile in a thread-safe manner. **Idempotency contract:** Calling `Unlock()` on an already-closed lock returns `nil`. This is intentional, not accidental. Error handling inside the method: If `l.closed == true` β†’ return `nil`. Close the file descriptor (`os.File.Close`) β€” error captured in local variable; if close fails, set `file = nil` so a subsequent close becomes a no-op. Remove the lockfile via `os.Remove`. If both close and remove fail: remove error overwrites prior close error only when close returned without error. If close failed first, any remove error is ignored. This means the final returned error reflects whichever operation last produced an error, not necessarily the most severe one β€” an intentional design choice worth flagging for callers. If `os.IsNotExist(err)` on remove β†’ silently accepted as normal idempotent cleanup.\n\n### Path Resolution\n\n**Signature:** `func SafeResolve(repoRoot, inputPath string) (string, error)`. **Responsibility:** Resolve any input path strictly inside the repository root. The function performs a multi-step containment check to defeat symlink-based escape vectors and physical directory traversal.\n\n**Algorithmic Steps:** (1) Call `filepath.Abs` on `inputPath`. (2) Evaluate symlinks on the existing ancestor via `EvalSymlinks` β€” prevents symlink-based escape from parent directories that exist but are symbolic links. (3) Walk up from the target until a physically-existing directory is found, then re-evaluate symlinks on that ancestor via `EvalSymlinks` again. (4) Rejoin with the remaining suffix components after the physical ancestor boundary. (5) Verify final result is under `resolvedRoot` via relative path check (`filepath.Rel`).\n\n**Error Handling:** Any error from `Abs`, first `EvalSymlinks`, second `EvalSymlinks`, or `Lstat` (non-exist) wraps with descriptive message using `fmt.Errorf(..., err)` β€” underlying cause recovered via `%w`. Unrecognized error from `Lstat` (`!os.IsNotExist`) returns immediately as wrapped error. Path escapes repo root (`..` prefix or non-nil rel) returns sentinel-style error wrapping `ErrPathTraversal` with the original input path.\n\n### Lock Acquisition\n\n**Signature:** `func AcquireLock(repoRoot string) (*SimpleLock, error)`. **Responsibility:** Establish an exclusive process-level lock on a shared resource. Uses `O_EXCL` flag for atomic creation of the lockfile β€” if the lockfile already exists, acquisition fails immediately without silent success.\n\n**Lockfile Lifecycle:** Resolve path calls `SafeResolve(repoRoot)` to obtain absolute lock path. Create atomically opens file with `O_WRONLY | O_CREATE | O_EXCL` β€” atomic exclusive creation prevents race between existence check and open. Record identity writes PID of the acquiring process to the newly created lockfile. Failure cleanup on failure: closes file, removes lockfile.\n\n**Error Handling:** `SafeResolve` fails propagates as-is to caller. File open fails AND error is `os.IsExist` (lock already held) returns wrapped sentinel error wrapping `ErrLockHeld`, including the lock path β€” caller can detect stale-lock condition specifically via `errors.Is(err, ErrLockHeld)`. File open fails with any other error wraps: `\"failed to acquire lock at %s: \u003ccause\u003e\"`. Write PID fails file closed, lockfile removed (cleanup), wrapped error returned.\n\n### Gitignore Maintenance\n\n**Signature:** `func EnsureGitignoreHasLockfile(repoRoot string) (error)`. **Responsibility:** Maintain the convention that the lockfile is excluded from version control but automatically tracked if missing. Appends the lockfile entry to `.gitignore` only if not already present; uses atomic rename via temp file + sync before close to prevent partial writes.\n\n**Algorithmic Steps:** (1) Resolve `.gitignore` path via `SafeResolve(repoRoot)`. (2) Read existing `.gitignore` content (`os.ReadFile`). (3) Parse lines, check if lockfile entry already exists. (4) If not present: construct new content with the lockfile entry appended (with trailing-newline normalization). (5) Create a temp file in same directory as `.gitignore`. (6) Write to temp file β†’ call `Sync()` on it β†’ close it β†’ `Chmod` it. (7) Rename temp file over original `.gitignore`.\n\n**Error Handling:** `SafeResolve` fails propagates as-is. Read fails AND not `os.IsNotExist` (file exists but read error) wraps: `\"error reading .gitignore: \u003ccause\u003e\"` β€” caller must handle missing-file vs I/O-error distinction separately. CreateTemp fails wraps with descriptive message including `%w`. Write to temp file fails wraps; temp file is NOT cleaned up (defer handles close + remove, but write error returns before cleanup path executes). Sync fails wraps β€” note that `os.Chmod` and subsequent rename may have already run on a partially-written temp file if this error occurs late in the sequence. Close fails wraps. Chmod fails wraps. Rename (atomic swap) fails wraps with descriptive message.\n\nThe deferred function (`tmpFile.Close()` + `os.Remove(tmpName)`) runs regardless of return path. Because the function returns before the defer has a chance to execute in some failure paths, cleanup may not happen for intermediate states β€” but since we're using temp file with unique name and same-directory rename, the OS-level temp file is cleaned by `os.Remove` in the defer on any error return path. If *write* fails mid-stream before close/rename, the deferred Remove will clean up the partially-written temp file.\n\n### Data Flow Summary\n\n```\n[Initialize] β†’ [Ensure path safety for all repo-root operations (SafeResolve)] β†’ \n[If no process holds the lock] β†’ [Create lockfile with PID via O_EXCL (AcquireLock)] β†’ \n[Run business logic within locked session] β†’ \n[Append entry to .gitignore if needed (EnsureGitignoreHasLockfile)]\n```\n\nThe two non-obvious constraints are: (1) path resolution walks symlinked ancestors before rejoining suffix components, and (2) gitignore updates use atomic rename with explicit sync before close.\n\n**Key Patterns Observed Across the Module:** Error wrapping throughout β€” All functions use `fmt.Errorf(..., err)` with `%w`, making root-cause recovery possible via `errors.Is()`. Custom sentinel errors β€” At least two custom error variables referenced: `ErrPathTraversal` and `ErrLockHeld`. These enable typed error detection by callers without requiring unwrapping logic on this file's side. No panic anywhere β€” All failure paths return to caller with an error value. Idempotent operations deliberately silent β€” `Unlock()` on already-closed lock returns nil; missing `.gitignore` in EnsureGitignoreHasLockfile returns nil. These are design choices, not accidentals. Atomic rename pattern for .gitignore β€” Write to temp β†’ Sync β†’ Close β†’ Chmod β†’ Rename is the standard atomic-update pattern. The `Sync()` call before close is explicit and unusual (normally deferred), but here it's called eagerly before close in this implementation. No global package-level mutable state exists in either file β€” all state is either local to a function or encapsulated within the `SimpleLock` receiver, which is accessed only through its methods.\n\n---\n\n## Subsystem: internal/tools\n\n### Module Responsibility\n\nThe `internal/tools` package provides repository-scanning primitives for a codebase-analysis tool. It exposes two functional domains: **safe local file I/O** (`file_tools.go`) and **git subprocess orchestration** (`git_tools.go`). Together they enable the caller to discover source-code files, classify their content type, read/write artifacts safely, and validate that the target directory is a git work-tree before proceeding.\n\nData flow follows a two-phase pattern: (1) `VerifyGitRepo` (or an equivalent pre-flight check) confirms the working directory corresponds to a valid git repository by invoking `git rev-parse --is-inside-work-tree`. (2) Once validated, `DiscoverCodeFiles` walks the entire tree under that root, applying layered ignore rules and returning only text-source files. Both domains share these architectural constraints: **no panics**, **error wrapping** (stderr is preserved for diagnostics), and **no network/DB calls**. All I/O operates within a single local process or on the local filesystem.\n\n### File Operations\n\n**Safe Read / Write Primitive:** `ReadFileSafely` and `WriteFileSafely` implement TOCTOU-safe file operations using the temp-file + rename pattern. Shared sequence: (1) Resolve a virtual path to an absolute filesystem path via `SafeResolve`. (2) On write: create parent directories with `defaultDirPerm`; on read: no directory creation. (3) Open or create a temporary file in the target directory using `os.CreateTemp`. (4) Write content (for writes) or read from the resolved path (for reads). (5) Close and sync (writes only); close (reads only). (6) Rename temp into final destination for writes; return contents for reads.\n\n**Symlink detection:** Both paths verify the target is not a symlink via `os.Lstat`/`f.Stat`. Pre-open checks prevent TOCTOU swap attacks where an attacker replaces a real file with a symlink between resolution and open. Post-open checks catch any state changes after the initial open.\n\n**Error semantics:** `ReadFileSafely`: returns `nil, err` on failure; wraps each step with context strings (`failed to open file`, `failed to lstat`, etc.). Symlink detection yields an explicit \"security violation\" error rather than a generic I/O error. `WriteFileSafely`: all failure paths return wrapped errors (directory creation, temp write, sync, close, chmod, rename). No panic usage; no sentinel values. Deferred cleanup: Both functions defer `f.Close()` for reads and use deferred `os.Remove` in `WriteFileSafely` to clean up temp files if the rename fails.\n\n**Gitignore Handling:** **`LoadGitignore(repoRoot)`**: Opens `.gitignore` under `repoRoot`. If absent (`os.IsNotExist`), returns `nil, nil` β€” no error raised. Any other open/read failure is returned unwrapped. **`ShouldIgnoreFile(repoRoot, relPath, gitIgnore)`**: Applies layered rules: (1) Check against compiled `GitIgnore` patterns (user-supplied + `.gitignore`). (2) Reject any path component starting with `.` or ending with `.egg-info`, regardless of other rules. (3) If extension matches a known text-source language list, accept as code. (4) For unknown extensions: resolve absolute path and scan first 1024 bytes for null bytes; binaries return `true`.\n\n**Side effect note:** `ShouldIgnoreFile` opens the file on every invocation to classify unknown extensions β€” this is a real I/O side effect per call, not cached.\n\n### Code Discovery Pipeline\n\n**Input:** repository root + list of ignore patterns (strings). **Output:** slice of relative source-code paths.\n\n**Algorithmic flow:** (1) Compile user-supplied ignore patterns and `.gitignore` entries into a single `GitIgnore` object via `LoadGitignore`. (2) Walk the entire tree recursively using `filepath.WalkDir`. For each entry: Directory: if name starts with `.` or matches any compiled pattern, skip subtree entirely. File: delegate to `ShouldIgnoreFile`. (3) Collect passing files into result slice.\n\n**Walk error handling:** If a walk callback returns an error for a single entry, the error is printed to `stderr` and that item is skipped (`return nil`). The overall `err` from `WalkDir` propagates at the end β€” terminal fatal errors are returned; non-fatal entries are swallowed silently.\n\n### Binary Classification\n\n**IsBinaryFile:** Scans first 1024 bytes for null bytes to distinguish text source files from binary content. Returns `true` (ignored) on any I/O error other than EOF β€” this \"fail-closed\" assumption treats unreadable files as likely binary rather than propagating the actual error.\n\n**Constants:** `binaryDetectionBufSize` is lowercase and package-private; not exported.\n\n### Git Tools\n\n**External Command Execution (`RunGit`):** Spawns `git --no-pager` via `exec.Command` with any additional arguments from the caller. The subprocess runs within `repoRoot` as its working directory. Stdout/stderr are captured into in-memory `bytes.Buffer`; no file, network, or DB I/O occurs.\n\n**Failure handling:** On error, wraps the original with stderr content:\n```\nfmt.Errorf(\"git command failed: %v, stderr: %s\", err, trimmedErr)\n```\nReturns `(string, error)` β€” partial stdout is still delivered alongside the error. No panics.\n\n**Repository Validation (`VerifyGitRepo`):** Delegates to `RunGit` with subcommand `rev-parse --is-inside-work-tree`. If this invocation results in an error, returns a fixed message: `\"not a git repository (or any of the parent directories)\"`. The original stderr is discarded at this wrapper layer β€” normalized to a single sentinel-style message for downstream callers.\n\n### Concurrency \u0026 State Analysis\n\n**Mutable state:** None detected across both files. All constants are `const` (compile-time); all variables (`safePath`, `dir`, `tmpFile`, `tmpName`, `patterns`, `files`, `buf`, `cmd`, `stdout`, `stderr`, etc.) are function-local and never escape their defining scope. No package-level global state is modified.\n\n**Concurrency mechanisms:** None detected. No `sync.Mutex`, channels, `atomic` types, or other synchronization primitives. Shared mutable state does not exist to protect.\n\n### Error Handling Summary\n\n| Pattern | Where |\n|---|---|\n| Wrap-and-return (preserve stderr) | `RunGit` |\n| Normalize to sentinel message | `VerifyGitRepo` |\n| Sentinel + wrap with context | `ReadFileSafely`, `WriteFileSafely`, `LoadGitignore` |\n| Boolean return, resolve failures as \"ignored\" | `ShouldIgnoreFile` |\n| Swallow per-entry walk errors, propagate terminal | `DiscoverCodeFiles` |\n| Fail-closed (I/O error β†’ binary assumption) | `IsBinaryFile` |\n\n**No panics** observed across either file. All errors are returned to callers as Go values.", + "internal/config": "# Module: internal/config β€” Configuration Layer for Code Reducer Synthesis Pipeline\n\n## Responsibility\n\nThis module manages the lifecycle of project-level configuration for a multi-phase LLM analysis pipeline that generates technical documentation from source code. It defines three distinct concerns: (1) compile-time type and constant declarations, (2) file-system operations on `.code-reducer.yaml` with atomic write guarantees, and (3) multi-source value resolution across defaults, YAML, environment variables, and CLI flags.\n\n## Data Flow\n\nThe module's data flow is sequential: `config.go` declares the schema (`Config`, `ExtractionStep`) and compile-time constants; `io.go` provides read/write primitives against a fixed path derived from `cwd`; `resolve.go` composes these into a resolved `*Config` by merging across four priority tiers. No state persists between calls within this moduleβ€”each resolution produces a fresh `*Config` allocation whose fields are populated through the merge chain and returned to callers for consumption elsewhere in the application.\n\n---\n\n## Configuration Types and Defaults (`config.go`)\n\n### Exported Structs\n\n**`ExtractionStep`** β€” Carries per-phase identifiers:\n- `Name string` β€” Phase identifier (e.g., `\"API_SIGNATURES\"`, `\"BUSINESS_LOGIC\"`).\n- `Prompt string` β€” The prompt text handed to the LLM for that phase.\n\n**`Config`** β€” Aggregate configuration container referenced by all exported functions:\n- `ModelID string`\n- `OllamaBaseURL string`\n- `OllamaNumCtx int`\n- `DocsDir string`\n- `SystemPrompt string`, `ModuleSynthesisPrompt string`, `ArchitecturePrompt string`, `FileFactConsolidationPrompt string` β€” First-class prompt templates, all configurable via YAML.\n- `ExtractionSteps []ExtractionStep`\n- `Ignore []string`\n\n### Exported Constants\n\nEnvironment variable keys (`CodeReducerModelIDEnvKey`, `OllamaBaseURLEnvKey`, `OllamaNumCtxEnvKey`), default values for Ollama instance parameters (`OllamaDefaultBaseURL = \"http://localhost:11434\"`, `OllamaDefaultModelID = \"ornith:9b\"`, `OllamaDefaultNumCtx = 8192`), file-system defaults (`DefaultDocsDir = \"wiki\"`, `ConfigFileName = \".code-reducer.yaml\"`), and multiline prompt templates are all package-level constants.\n\n### Exported Variable\n\n**`DefaultExtractionSteps`** β€” Pre-populated slice containing four named phases: `API_SIGNATURES`, `BUSINESS_LOGIC`, `STATE_AND_CONCURRENCY`, `ERRORS_AND_SIDE_EFFECTS`. Initialized at declaration only; no post-initialization mutations occur in this file.\n\n### Non-Exported\n\nInternal helper variable `configFilePerm` (value `0600`) is referenced by `io.go` but not declared here. No exported interfaces or methods exist on these types.\n\n---\n\n## Configuration File Lifecycle (`io.go`)\n\n### Exported Functions\n\n| Function | Input | Output |\n|----------|-------|--------|\n| `ConfigExists(cwd string)` | cwd directory path | `bool` |\n| `LoadConfig(cwd string)` | cwd directory path | `*Config`, `error` |\n| `SaveConfig(cwd string, cfg *Config)` | cwd, config pointer | `error` |\n\nNo exported structs or interfaces are defined in this file. All external type references (`Config`) come from the same package.\n\n### Atomic Write Pattern for SaveConfig\n\nThe save operation does not write directly to the target path. The sequence is: marshal `cfg` into YAML bytes β†’ apply formatting normalization (collapsing single blank lines to double blank lines before known section headers: `system_prompt`, `module_synthesis_prompt`, `architecture_prompt`, `file_fact_consolidation_prompt`, `extraction_steps`, `ignore`) β†’ create a temp file matching `.tmp.*` in the same directory β†’ write normalized YAML into it β†’ sync and close β†’ set permissions via `os.Chmod` β†’ atomically rename temp over original. Readers never observe a partially-written config during save because the old file remains intact until the new one is fully ready to replace it.\n\n### Load-Save Roundtrip Integrity\n\nLoading preserves whatever formatting was originally written; normalization is applied only on write, not read. This decouples the in-memory representation from on-disk formatting expectations. `LoadConfig` returns a fresh `*Config` via unmarshaling with no shared state. Errors from `os.ReadFile` propagate directly; YAML parse failures are wrapped with a `\"failed to parse yaml config\"` prefix chained via `%w`; marshal failures similarly wrap a descriptive message.\n\n### Error Propagation Strategy\n\n- **Raw return**: When `os.ReadFile` fails in `LoadConfig`, the OS error is returned unwrapped.\n- **Wrap with context (`fmt.Errorf(... %w)`)**: Parse and write failures across all save-path operations (temp create, write, sync, close, chmod, rename). All wrapped calls preserve chain via `%w`.\n- **Error-as-boolean sentinel**: `ConfigExists` uses an `err == nil` check instead of returning the error. No wrapping; raw error is reused as a boolean condition.\n- **Silent cleanup in defer**: A deferred function calls `tmpFile.Close()` and `os.Remove(tmpName)`. Errors from these cleanup calls are swallowed (not returned). On success, `Close` runs twice on the temp fileβ€”Go allows this safelyβ€”and the defer's `Remove` returns an error that is silently swallowed because the original config now holds the renamed data.\n\n### Disk Operations Summary\n\n| Function | Disk Operations |\n|---|---|\n| `getConfigPath` (internal) | None; pure computation via `filepath.Join`. |\n| `ConfigExists` | One read-only check via `os.Stat`. Error sentinel reused as boolean condition. |\n| `LoadConfig` | Two disk reads: `os.ReadFile` then `yaml.Unmarshal`. Missing-file errors propagate raw; parse failures are wrapped. |\n| `SaveConfig` | Multiple writes: `os.CreateTemp`, `tmpFile.Write`, `tmpFile.Sync`, `tmpFile.Close`, `os.Chmod`, `os.Rename`. Each step returns its own error; all but the final rename are wrapped. |\n\nNo network I/O anywhere in this module. No database operations referenced.\n\n---\n\n## Multi-Source Configuration Resolution (`resolve.go`)\n\n### Exported Functions\n\n| Function | Input Types | Output Types |\n|----------|-------------|--------------|\n| `ResolveConfig(repoRoot, modelIDFlag, numCtxFlag string)` | `(string, string, string)` | `(*config.Config, error)` |\n\nInternal helper `mergeAndDeduplicate` is lowercase and not part of the public surface. Only usage (not definition) of package-level types (`Config`, `OllamaDefaultModelID`, `CodeReducerModelIDEnvKey`, `DefaultExtractionSteps`) is visible here.\n\n### Resolution Priority System\n\nEach configurable field follows its own independent priority chain, lowest to highest: **defaults β†’ YAML config file β†’ environment variables β†’ CLI flags**. The merge-and-deduplicate logic operates on local function scopes with no cross-goroutine visibility and no shared mutable state.\n\n### Field-Specific Priority Chains\n\n| Field | Priority Chain |\n|-------|---------------|\n| `ModelID` | Default \u003e YAML \u003e Env Var \u003e CLI Flag |\n| `OllamaBaseURL` | Default \u003e YAML \u003e Env Var |\n| `OllamaNumCtx` (context size) | Default \u003e YAML \u003e Env Var \u003e CLI Flag |\n| `DocsDir` | Default \u003e YAML |\n| System prompts (`SystemPrompt`, `ModuleSynthesisPrompt`, `ArchitecturePrompt`, `FileFactConsolidationPrompt`) | All start with default, then override with YAML if non-empty |\n\n### Validation Rules\n\n- **Numeric fields**: CLI flag and env var values are parsed as integers; a value must be greater than zero to take effect. Invalid or non-positive values fall through silently.\n- **String fields**: Empty string is treated as \"not set\" β€” the lower-priority default remains active.\n- **Config file absence** (`config.yml`): Not an error. If missing, an empty `Config{}` struct is created and defaults are applied. Other parse errors are surfaced.\n\n### Algorithmic Flow for ResolveConfig\n\n1. Load YAML config β€” if it exists and parses successfully; otherwise start with an empty struct (missing-file case).\n2. Resolve extraction steps β€” use YAML value if present, else fall back to `DefaultExtractionSteps`.\n3. Deduplicate ignore list from the loaded config using a map-based seen-set while preserving first-seen order.\n4. For each field, apply its specific priority chain: start with built-in default constant β†’ override with YAML (if non-empty) β†’ override with environment variable (if set and valid for numeric types) β†’ override with CLI flag (highest priority; numeric types require successful integer parse and positive value).\n5. Return the fully resolved `Config` struct, or an error if the YAML file fails to parse (excluding missing-file case).\n\n### Error Handling in ResolveConfig\n\n- **Swallowed path**: When `os.IsNotExist(err)` is true during config loading, defaults apply with no error returned.\n- **Propagated path**: Any other error (parse failure, permission denied) is wrapped with a prefix and returned as `(*Config, nil, error)` β€” the config pointer is `nil`.\n- **Silent branches**: String comparisons (`!= \"\"`, `\u003e 0`) short-circuit without errors. `strconv.Atoi` failures are caught inline; env/flag values simply not used. Map lookups for deduplication have no error path.\n\n### Always-On Return\n\nThe function always returns a non-nil tuple `(*Config, error)` β€” either a populated config with `nil` error, or a `nil` config with an error string. No panics anywhere in this file.\n\n---\n\n## Concurrency and Mutability Summary\n\nAcross all three files: no `sync.Mutex`, channels, atomics, or other synchronization primitives appear. No package-level variables are modified after initialization. All functions operate on caller-provided arguments and return results. The module is designed for single-call composition with no shared mutable state requiring protection.", + "internal/engine": "# internal/engine β€” Documentation Synthesis Engine\n\n## Module Responsibility \u0026 Data Flow\n\nThe `internal/engine` package implements a recursive, context-window-aware LLM-based documentation synthesis engine. It operates in two modesβ€”**init** (full generation from scratch) and **update** (incremental refresh)β€”orchestrating file discovery, change detection, chunked fact extraction, hierarchical synthesis, and persistent cache management for all intermediate state.\n\nData flow: `Runner.Run` β†’ acquires lock β†’ instantiates LLM client + orchestrator β†’ dispatches to `RunInit` or `RunUpdate` β†’ discovers code files via `tools.DiscoverCodeFiles` β†’ classifies changes (Added/Modified/Deleted) against persisted `MetadataCache` β†’ builds directory tree β†’ propagates affected flags upward β†’ recursively synthesizes each node via `synthesizeNode` (per-chunk LLM extraction β†’ fact consolidation β†’ reduction) β†’ writes module docs to disk β†’ persists cache.\n\nAll external communication is delegated: network calls go through the injected `llmCaller.CallLLM`; filesystem I/O goes through internal tool wrappers (`tools.ReadFileSafely`, `tools.WriteFileSafely`); path resolution uses `security.SafeResolve`. No mutexes, channels, or atomic operations are used anywhere in this package.\n\n---\n\n## Constants \u0026 Types (constants.go)\n\n### Constants\n\n| Constant | Value | Purpose |\n|---|---|---|\n| `defaultHTTPTimeout` | 10 min | Default HTTP client request timeout |\n| `maxErrorBodyBytes` | 1024 bytes | Maximum size allowed in error response bodies |\n| `defaultChunkOverlap` | 800 chars | Overlap between consecutive chunks during streaming/processing |\n| `minNumCtxFloor` | 512 tokens | Minimum context window floor for processing |\n| `contextWindowAllocRatio` | 0.75 | Ratio used when allocating context windows (reserves 75% of available space) |\n| `maxCharsMultiplier` | 3 | Multiplier applied to character counts (scales token estimates or max output limits) |\n| `metadataFileName` | `.metadata.json` | Persistent metadata file name for the engine's state |\n| `agentsFileName` | `AGENTS.md` | Agent definitions/configuration file |\n| `defaultDirPerm` | 0755 | Default directory permission mode |\n\n### Type: `LogEventFunc`\n\nFunction type alias with signature `func(EventType, string)`. Callback interface for logging events; no business rules defined here.\n\n---\n\n## LLM Client Communication (client.go)\n\n### `Message`, `llmClient`, `CallLLM`\n\nUnexported except `Message` struct (`Role string`, `Content string`). The package-private `llmClient` carries: model ID, base URL, context count, HTTP client pointer. All set once during construction; never modified afterward within this file. No mutexes or channels used.\n\n### Algorithmic Flow\n\n1. **Prepare payload**: Assemble request body by prepending `systemPrompt` as a system-role message before any user-supplied messages, serialize as JSON to Ollama `/api/chat`.\n2. **Send HTTP request**: Single synchronous POST with fixed timeout; no retries (fail-fast policy).\n3. **Handle response**: On 200 OK β†’ deserialize body β†’ extract `message.content` field. Non-2xx status code β†’ error string containing both HTTP status and raw response payload.\n4. **Return result**: Extracted text content or wrapped error describing what went wrong.\n\n### Error Handling Patterns\n\n| Path | Wraps? | Notes |\n|---|---|---|\n| Marshal failure | βœ… `%w` | Good; caller can inspect via `errors.Is` |\n| HTTP request construction | ❌ raw | Unwrapped; context lost for callers |\n| Network/transport error | ❌ raw | Unwrapped; timeout vs DNS vs connect indistinguishable to caller |\n| Success-path body read | ❌ raw | Unwrapped (rare) |\n| JSON unmarshal failure | βœ… `%w` | Good |\n| Non-OK status response | ❌ custom message | Discards read error (`_`); uses undefined `maxErrorBodyBytes` constant |\n\nNo retry logic. `defaultHTTPTimeout` and `maxErrorBodyBytes` referenced but not defined in this file; if they come from another package, callers have no visibility into their values or how they are set. No context propagation check between `Do()` and `Body.Close()`.\n\n---\n\n## Runner \u0026 Entry Point (runner.go)\n\n### `Runner` / `Run`\n\n`Runner` wraps a config pointer and exposes a single `Run(ctx, repoRoot, mode, onEvent)` method that is the only public entry point into the pipeline. It performs four sequential operations:\n\n1. **Gitignore lockfile maintenance** β€” attempts to append the lockfile path to `.gitignore`. Failure is logged via `onEvent(EventStatus, ...)` and execution continues; no error returned.\n2. **Repository lock acquisition** β€” calls `security.AcquireLock(repoRoot)`. On failure, returns wrapped error: `\"failed to acquire repository lock: %w\"`. No partial work proceeds without the lock.\n3. **Mode dispatch** β€” constructs an LLM client from config fields (`cfg.ModelID`, `cfg.OllamaBaseURL`, `cfg.OllamaNumCtx`) and instantiates an `orchestrator`. Dispatches to either `RunInit` or `RunUpdate`; any other mode returns `\"unsupported mode: %s\"`.\n4. **Deferred lock release** β€” `defer lock.Unlock()` runs after the pipeline completes, regardless of success or failure.\n\nThe `Runner.cfg` field is set once in `NewRunner(cfg)` and never mutated within this file. The pointer type allows external reassignment but no such mutation occurs here.\n\n---\n\n## Orchestrator \u0026 Pipeline Modes (orchestrator.go)\n\n### `EventType`, `Event`, `orchestrator`\n\nUnexported struct with two exported constants on the `EventType` type alias: `EventStatus = \"status\"` and `EventError = \"error\"`. The unexported `orchestrator` struct carries a method receiver; its public surface is empty.\n\n### RunInit β€” Cold Start Flow\n\n```\n1. setupPipeline() β†’ discover files, compute hashes, load .gitignore (swallowed), load cache (swallowed)\n2. Mark every directory as affected (full regeneration)\n3. Synthesize root summary from full tree via LLM calls\n4. Generate architecture.md + quickstart.md using root summary\n5. Write agent guidelines file (append if marker missing, overwrite otherwise)\n6. Persist cache to disk β†’ complete\n```\n\n### RunUpdate β€” Incremental Refresh Flow\n\n```\n1. setupPipeline() β†’ discover current files, compute hashes\n2. If extraction steps changed β†’ invalidate entire cache (.StepsHash mismatch resets Files and Modules maps)\n3. Classify all discovered files into Added/Modified/Deleted buckets by comparing current vs cached SHA-256 hashes\n4. Build directory tree from allowed (existing + new) files only\n5. Purge stale module summaries: for any `cache.Modules` path not represented in the live tree, delete both the cache entry and the physical `.md` file (`_ = os.Remove(...)` β€” error swallowed).\n6. Compute affected directories; propagate upward through parent dirs.\n7. If `len(affectedDirs) == 0` β†’ return early (no-op, docs are up to date).\n8. Synthesize root summary from the affected region via LLM calls.\n9. Re-generate architecture.md + quickstart.md only if `.` is in `affectedDirs`, or either file does not exist on disk, or resolution fails (treated as \"not exists\").\n10. Persist cache β†’ complete.\n```\n\n### Pipeline Context (`pipelineContext`)\n\nUnexported struct carrying: LLM client pointer, repo root string, config pointer, `*MetadataCache` pointer, affected directory set map, precomputed file hashes map, and event logger callback. Constructed fresh per invocation; no shared instances exist across goroutines in this file.\n\n---\n\n## Cache Persistence Layer (cache.go)\n\n### `MetadataCache`, `FileCacheEntry`\n\nTwo structs with no methods:\n- **`FileCacheEntry`** β€” pairs a SHA256 digest string (`sha256`) with extracted facts string (`facts`). Stored in `MetadataCache.Files map[string]FileCacheEntry`.\n- **`MetadataCache`** β€” carries version int, steps hash string, files map, and modules map (module dir β†’ synthesized summary).\n\n### IsInitialized(repoRoot, docsDir) bool\n\nReturns true only if the cache file exists AND is readable in one call. Reuses the existence check from `loadMetadataCache`. No error return.\n\n### saveMetadataCache(repoRoot, docsDir, cache) error\n\nSerializes to `{docsDir}/{metadataFileName}` with indented JSON (`json.MarshalIndent(cache, \"\", \" \")`). Marshaling errors propagate directly; write errors pass through `tools.WriteFileSafely`. The version field is pinned to the current value on every save.\n\n### Version Compatibility Guard\n\nOn load: if `cache.Version != currentCacheVersion` (currently 1), returns a fresh zero-valued `MetadataCache{}` with nil error. No warning, no panicβ€”caller sees success with an empty cache. This prevents stale metadata from corrupting future runs without explicit migration handling.\n\n### Fault-Tolerant Initialization\n\nIf the file does not exist (`os.ErrNotExist` sentinel matched via `errors.Is`), returns a populated zero-valued cache with nil error. If it exists but cannot be read or unmarshaled, returns wrapped error `\"failed to read metadata cache: %w\"`. Callers can distinguish \"not yet initialized\" from \"corrupt data.\"\n\n---\n\n## Chunking \u0026 Reduction Engine (chunking.go)\n\n### reduceWithLLM, reduceInChunks, reduceFileFacts\n\nAll unexported; package-private. Implements a **context-aware recursive reduction** for LLM-based synthesis:\n\n1. **Split**: Input list divided into batches whose combined character count fits within `contextSize Γ— maxCharsMultiplier`.\n2. **Reduce each batch**: Each batch sent to LLM via domain-specific prompt β†’ one summary string per batch. If only one batch exists, returned directly; otherwise summaries concatenated and fed back into step 1 recursively until a single result remains.\n3. **Graceful truncation**: Items exceeding `maxChars` are truncated with `\"...\\n...[truncated]\"`. If every item exceeds the per-item budget (`allowedPerItem`), all items uniformly truncated to avoid infinite loops.\n4. **Context cancellation**: Checked at entry points and before each recursive reduction round; cancellation errors propagate as-is.\n\n### chunkTextWithOverlap (unused in this file)\n\nText-splitting utility with overlap between adjacent chunks. Appears available for reuse elsewhere but not referenced by any function here.\n\n---\n\n## Synthesis Core (synthesize.go)\n\nAll types and functions unexported (`pipelineContext`, `synthesizeNode`). Package-private.\n\n### Recursive Directory Synthesis\n\nA directory's summary is derived by recursively synthesizing children (subdirectories) and files together. Final result stored in cache under `cache.Modules[node.Path]` and written to disk at `docs/modules/\u003csafe-filename\u003e`.\n\n### Caching Strategy with Hash-Based Invalidation\n\n- **Module-level**: Stores synthesized summary per directory path. Reused when affected status indicates no changes (short-circuit return).\n- **File-level**: Stores SHA256 hash + extracted facts in `cache.Files[f]`. Cache hit determined by comparing precomputed or read file hash against cached entry; only re-reads on miss.\n\n### Multi-Step Extraction Pipeline Per File Chunk\n\n1. System prompt augmented with current extraction step's prompt.\n2. Single LLM call extracts facts from chunk β†’ one fact per step.\n3. All steps produce respective facts for the same file β†’ consolidated via `reduceFileFacts`.\n\n### Synthesis Order \u0026 Assembly\n\n1. File facts (sorted child name order).\n2. Child subdirectory summaries (after all files processed).\n3. Combined list reduced to final summary via `reduceInChunks` at directory level.\n\n### Affected-Dir Pruning\n\nIf a directory path has no affected status AND a cached module summary exists β†’ short-circuit return cached value without reprocessing children or files.\n\n---\n\n## Tree \u0026 Change Detection (tree.go)\n\n### Exported Types: `FileChange`, `DirNode`\n\n- **`FileChange`** β€” `Path string`, `Status string` (\"Added\", \"Modified\", \"Deleted\").\n- **`DirNode`** β€” `Path string`, `Files []string`, `Children map[string]*DirNode`.\n\n### Three-Phase Algorithm\n\n1. **Build tree**: Given raw file paths, split each on `/`, create intermediate `DirNode` entries for subdirectories, attach files to leaf nodes via `buildTree`.\n2. **Determine affected directories from changes**: For each input change, mark directly changed file as affected (if exists in current set). When a file is Deleted, mark parent directory as affected. Walk recursively and check additional conditions: if node path matches `docs/modules/\u003csafe-name\u003e` but no such doc exists on disk β†’ affected; if node has empty cached metadata entry β†’ affected.\n3. **Propagate status upward**: Recursive walk of `DirNode` tree propagates any `affected` flag from children to parents so a parent is considered affected if any child is.\n\n---\n\n## JSON Parser Utility (json_parser.go)\n\n### stripOuterMarkdownFence(input) string\n\nSingle-pass parser that strips markdown or JSON code fences from strings:\n1. Trims whitespace from input.\n2. Attempts regex match against pattern capturing triple backticks with 3+ chars, optionally followed by `markdown` or `json`, then all content between opening and closing fences (group 1).\n3. If matched β†’ returns trimmed captured inner content.\n4. If no match β†’ returns original trimmed input unchanged.\n\nNo error return parameter. `regexp.MustCompile` panics on invalid pattern at package initβ€”unhandled within this file, propagates to any caller importing the package.\n\n---\n\n## Utility Functions for Filename Generation and Event Logging (utils.go)\n\n### Responsibility\n\nThis file provides two unexported utility functions supporting the engine's documentation generation pipeline and optional event logging instrumentation. Neither function communicates with external systems, modifies shared state, or returns errors. All operations are pure in-memory transformations.\n\n---\n\n### toSafeMarkdownFilename β€” Module Path to Markdown Filename Conversion\n\n**Signature:** `func toSafeMarkdownFilename(path string) string` (inferred from usage; exact signature not exported)\n\n#### Purpose\n\nTransforms a module path into a safe markdown documentation filename, enforcing the engine's implicit one-to-one mapping contract between modules and their `.md` documentation files. The root module is special-cased to default to `root.md`.\n\n#### Data Flow\n\n1. Input: a raw string representing a module path (e.g., `\"internal/engine\"`).\n2. Character substitution: every `/` character in the input is replaced with `_`.\n3. Extension appended: `.md` is concatenated to produce the final filename.\n4. Output: a single `string` containing the resulting markdown filename.\n\n#### Behavior Notes\n\n- No filesystem, network, database, or other external I/O occurs during execution.\n- The function returns only a result string; no error type is returned. Invalid input characters (anything other than `/`) are passed through unmodified β€” callers receive no escape hatch for detecting malformed paths.\n- Pure: no mutable state, no package-level globals modified, no struct fields touched.\n\n---\n\n### makeLogEvent β€” Optional Event Callback Adapter\n\n**Signature:** `func makeLogEvent(onEvent LogEventFunc) func(EventType, message string)` (inferred from usage; `EventType`, `LogEventFunc` referenced but not defined in this file)\n\n#### Purpose\n\nWires an optional callback into a logging pipeline. When invoked with an event type and message pair, it either forwards the call to the registered handler or silently discards it. This makes the logging subsystem purely additive β€” no required consumer exists for the engine's log events; instrumentation is opt-in only.\n\n#### Data Flow\n\n1. Input: a user-provided `onEvent` callback of type `LogEventFunc`.\n2. Closure construction: returns a new function accepting `(EventType, message)` pairs.\n3. Invocation behavior: if `onEvent == nil`, the returned closure does nothing when called β€” no panic, no error, no log output. If `onEvent` is non-nil, the call is forwarded to it with the provided type and message arguments.\n\n#### Behavior Notes\n\n- No I/O anywhere in this flow.\n- Pure: no mutable state, no package-level globals modified.\n- Silent failure path: when `onEvent == nil`, events are swallowed without notice or error wrapping.\n- The returned function is always valid (non-nil), so callers can invoke it safely regardless of whether a handler was registered.\n\n---\n\n### Referenced Types (Defined Elsewhere)\n\nThe following types are referenced in this file but not defined within it:\n\n| Type | Role |\n|---|---|\n| `EventType` | Categorizes log events into discrete types; carried as the first argument to event handlers and consumed by the returned closure from `makeLogEvent`. |\n| `LogEventFunc` | Function signature for user-provided event handlers. Expected to accept `(EventType, message string)` or equivalent parameters. |\n| `Event` (struct) | Domain model with at least two fields: `Type` (`EventType`) and `Message` (`string`). Events are tracked as discrete, typed records carrying descriptive payloads. |\n\n---\n\n### Concurrency \u0026 State Summary\n\n- **Mutable state:** None. Both functions operate entirely on their inputs and return values without touching package-level variables or struct fields.\n- **Concurrency mechanisms:** None. No mutexes, channels, atomic operations, or synchronization primitives are used in this file.", + "internal/security": "# internal/security β€” Path Containment \u0026 Process Mutual Exclusion\n\n## Module Responsibility\n\nThis module enforces two non-negotiable invariants for any code-reducer session: (1) the working directory must remain strictly within the repository root boundary, and (2) only one process may hold an exclusive lock on the shared resource at any given moment. It also maintains a `.gitignore` entry that excludes the lockfile from version control while ensuring it is automatically tracked if absent.\n\nThe module communicates with the outside world indirectly: `security.go` performs filesystem I/O and returns sentinel error values to callers; those sentinels are defined in `errors.go`. No panic recovery, no error wrapping beyond `%w`, no silent swallowing of non-trivial failures occurs within these files.\n\n## Sentinel Error Contract (`errors.go`)\n\n### Package-Level Variables\n\n| Identifier | Type | Purpose |\n|---|---|---|\n| `ErrPathTraversal` | `error` | Returned when a resolved path escapes outside the repository root boundary |\n| `ErrLockHeld` | `error` | Returned when another code-reducer process holds an exclusive file lock on the shared resource |\n\nBoth variables are initialized once via `errors.New` and remain read-only throughout their scope. No mutable state, no concurrency primitives, no panic handling exists in this file. Callers detect violations by matching against these sentinels using `errors.Is(err, ErrPathTraversal)` or equivalent patterns; because the errors are unwrapped as-is (no wrapping), identity is preserved directly through the error chain.\n\n### Domain Concepts\n\n- **Path Traversal Violation** (`ErrPathTraversal`) β€” signals when a resolved path escapes outside the repository root boundary. This is a classic path traversal / directory escape guardrail.\n- **Lock Conflict** (`ErrLockHeld`) β€” signals when another process already holds an exclusive file lock, indicating contention on a shared resource.\n\n## Lock Acquisition and Release (`security.go` β€” `SimpleLock`)\n\n### Struct Definition\n\n```go\ntype SimpleLock struct {\n lockPath string\n file *os.File\n mu sync.Mutex\n closed bool\n}\n```\n\n| Field | Type | Semantics |\n|---|---|---|\n| `lockPath` | `string` | Absolute path to the lockfile on disk |\n| `file` | `*os.File` | Open file descriptor for the lockfile, or nil if not acquired |\n| `mu` | `sync.Mutex` | Protects internal struct fields during `Unlock()` |\n| `closed` | `bool` | Marks whether the lock has been released and the file descriptor closed |\n\n### `*SimpleLock.Unlock()`\n\nCloses the lock file descriptor and removes the lockfile in a thread-safe manner.\n\n**Idempotency contract:** Calling `Unlock()` on an already-closed lock returns `nil`. This is intentional, not accidental.\n\n**Error handling inside the method:**\n- If `l.closed == true` β†’ return `nil`\n- Close the file descriptor (`os.File.Close`) β†’ error captured in local variable; if close fails, set `file = nil` so a subsequent close becomes a no-op\n- Remove the lockfile via `os.Remove`\n- If *both* close and remove fail: remove error overwrites prior close error only when close returned without error. If close failed first, any remove error is ignored. This means the final returned error reflects whichever operation last produced an error, not necessarily the most severe one β€” an intentional design choice worth flagging for callers.\n- If `os.IsNotExist(err)` on remove β†’ silently accepted as normal idempotent cleanup\n\n## Path Resolution (`security.go` β€” `SafeResolve`)\n\n### Signature\n\n```go\nfunc SafeResolve(repoRoot, inputPath string) (string, error)\n```\n\n**Responsibility:** Resolve any input path strictly inside the repository root. The function performs a multi-step containment check to defeat symlink-based escape vectors and physical directory traversal.\n\n### Algorithmic Steps\n\n1. Call `filepath.Abs` on `inputPath`\n2. Evaluate symlinks on the existing ancestor via `EvalSymlinks` β€” prevents symlink-based escape from parent directories that exist but are symbolic links\n3. Walk up from the target until a physically-existing directory is found, then re-evaluate symlinks on that ancestor via `EvalSymlinks` again\n4. Rejoin with the remaining suffix components after the physical ancestor boundary\n5. Verify final result is under `resolvedRoot` via relative path check (`filepath.Rel`)\n\n### Error Handling\n\n| Condition | Handling |\n|---|---|\n| Any error from `Abs`, first `EvalSymlinks`, second `EvalSymlinks`, or `Lstat` (non-exist) | Wrapped with descriptive message using `fmt.Errorf(..., err)` β€” underlying cause recovered via `%w` |\n| Unrecognized error from `Lstat` (`!os.IsNotExist`) | Returned immediately as wrapped error |\n| Path escapes repo root (`..` prefix or non-nil rel) | Returns sentinel-style error wrapping `ErrPathTraversal` with the original input path |\n\n## Lock Acquisition (`security.go` β€” `AcquireLock`)\n\n### Signature\n\n```go\nfunc AcquireLock(repoRoot string) (*SimpleLock, error)\n```\n\n**Responsibility:** Establish an exclusive process-level lock on a shared resource. Uses `O_EXCL` flag for atomic creation of the lockfile β€” if the lockfile already exists, acquisition fails immediately without silent success.\n\n### Lockfile Lifecycle\n\n| Phase | Mechanism |\n|---|---|\n| Resolve path | Calls `SafeResolve(repoRoot)` to obtain absolute lock path |\n| Create atomically | Opens file with `O_WRONLY \\| O_CREATE \\| O_EXCL` β€” atomic exclusive creation prevents race between existence check and open |\n| Record identity | Writes PID of the acquiring process to the newly created lockfile |\n| Failure cleanup | On failure: closes file, removes lockfile |\n\n### Error Handling\n\n| Condition | Handling |\n|---|---|\n| `SafeResolve` fails | Propagated as-is to caller |\n| File open fails AND error is `os.IsExist` (lock already held) | Returns wrapped sentinel error wrapping `ErrLockHeld`, including the lock path β€” caller can detect stale-lock condition specifically via `errors.Is(err, ErrLockHeld)` |\n| File open fails with any other error | Wrapped: `\"failed to acquire lock at %s: \u003ccause\u003e\"` |\n| Write PID fails | File closed, lockfile removed (cleanup), wrapped error returned |\n\n## Gitignore Maintenance (`security.go` β€” `EnsureGitignoreHasLockfile`)\n\n### Signature\n\n```go\nfunc EnsureGitignoreHasLockfile(repoRoot string) (error)\n```\n\n**Responsibility:** Maintain the convention that the lockfile is excluded from version control but automatically tracked if missing. Appends the lockfile entry to `.gitignore` only if not already present; uses atomic rename via temp file + sync before close to prevent partial writes.\n\n### Algorithmic Steps\n\n1. Resolve `.gitignore` path via `SafeResolve(repoRoot)`\n2. Read existing `.gitignore` content (`os.ReadFile`)\n3. Parse lines, check if lockfile entry already exists\n4. If not present: construct new content with the lockfile entry appended (with trailing-newline normalization)\n5. Create a temp file in same directory as `.gitignore`\n6. Write to temp file β†’ call `Sync()` on it β†’ close it β†’ `Chmod` it\n7. Rename temp file over original `.gitignore`\n\n### Error Handling\n\n| Condition | Handling |\n|---|---|\n| `SafeResolve` fails | Propagated as-is |\n| Read fails AND not `os.IsNotExist` (file exists but read error) | Wrapped: `\"error reading .gitignore: \u003ccause\u003e\"` β€” caller must handle missing-file vs I/O-error distinction separately |\n| CreateTemp fails | Wrapped with descriptive message including `%w` |\n| Write to temp file fails | Wrapped; temp file is NOT cleaned up (defer handles close + remove, but write error returns before cleanup path executes) |\n| Sync fails | Wrapped β€” note that `os.Chmod` and subsequent rename may have already run on a partially-written temp file if this error occurs late in the sequence |\n| Close fails | Wrapped |\n| Chmod fails | Wrapped |\n| Rename (atomic swap) fails | Wrapped with descriptive message |\n\nThe deferred function (`tmpFile.Close()` + `os.Remove(tmpName)`) runs regardless of return path. Because the function returns before the defer has a chance to execute in some failure paths, cleanup may not happen for intermediate states β€” but since we're using temp file with unique name and same-directory rename, the OS-level temp file is cleaned by `os.Remove` in the defer on any error return path. If *write* fails mid-stream before close/rename, the deferred Remove will clean up the partially-written temp file.\n\n## Data Flow Summary\n\n```\n[Initialize] β†’ [Ensure path safety for all repo-root operations (SafeResolve)] β†’ \n[If no process holds the lock] β†’ [Create lockfile with PID via O_EXCL (AcquireLock)] β†’ \n[Run business logic within locked session] β†’ \n[Append entry to .gitignore if needed (EnsureGitignoreHasLockfile)]\n```\n\nThe two non-obvious constraints are: (1) path resolution walks symlinked ancestors before rejoining suffix components, and (2) gitignore updates use atomic rename with explicit sync before close.\n\n## Key Patterns Observed Across the Module\n\n1. **Error wrapping throughout** β€” All functions use `fmt.Errorf(..., err)` with `%w`, making root-cause recovery possible via `errors.Is()`.\n2. **Custom sentinel errors** β€” At least two custom error variables referenced: `ErrPathTraversal` and `ErrLockHeld`. These enable typed error detection by callers without requiring unwrapping logic on this file's side.\n3. **No panic anywhere** β€” All failure paths return to caller with an error value.\n4. **Idempotent operations deliberately silent** β€” `Unlock()` on already-closed lock returns nil; missing `.gitignore` in EnsureGitignoreHasLockfile returns nil. These are design choices, not accidentals.\n5. **Atomic rename pattern for .gitignore** β€” Write to temp β†’ Sync β†’ Close β†’ Chmod β†’ Rename is the standard atomic-update pattern. The `Sync()` call before close is explicit and unusual (normally deferred), but here it's called eagerly before close in this implementation.\n6. **No global package-level mutable state** exists in either file β€” all state is either local to a function or encapsulated within the `SimpleLock` receiver, which is accessed only through its methods.", + "internal/tools": "# Package: internal/tools β€” Architecture Document\n\n## Module Responsibility\n\nThe `internal/tools` package provides repository-scanning primitives for a codebase-analysis tool. It exposes two functional domains: **safe local file I/O** (`file_tools.go`) and **git subprocess orchestration** (`git_tools.go`). Together they enable the caller to discover source-code files, classify their content type, read/write artifacts safely, and validate that the target directory is a git work-tree before proceeding.\n\nData flow follows a two-phase pattern:\n1. `VerifyGitRepo` (or an equivalent pre-flight check) confirms the working directory corresponds to a valid git repository by invoking `git rev-parse --is-inside-work-tree`.\n2. Once validated, `DiscoverCodeFiles` walks the entire tree under that root, applying layered ignore rules and returning only text-source files.\n\nBoth domains share these architectural constraints: **no panics**, **error wrapping** (stderr is preserved for diagnostics), and **no network/DB calls**. All I/O operates within a single local process or on the local filesystem.\n\n---\n\n## File Operations (`file_tools.go`)\n\n### Safe Read / Write Primitive\n\n`ReadFileSafely` and `WriteFileSafely` implement TOCTOU-safe file operations using the temp-file + rename pattern.\n\n**Shared sequence:**\n1. Resolve a virtual path to an absolute filesystem path via `SafeResolve`.\n2. On write: create parent directories with `defaultDirPerm`; on read: no directory creation.\n3. Open or create a temporary file in the target directory using `os.CreateTemp`.\n4. Write content (for writes) or read from the resolved path (for reads).\n5. Close and sync (writes only); close (reads only).\n6. Rename temp into final destination for writes; return contents for reads.\n\n**Symlink detection:** Both paths verify the target is not a symlink via `os.Lstat`/`f.Stat`. Pre-open checks prevent TOCTOU swap attacks where an attacker replaces a real file with a symlink between resolution and open. Post-open checks catch any state changes after the initial open.\n\n**Error semantics:**\n- `ReadFileSafely`: returns `nil, err` on failure; wraps each step with context strings (`failed to open file`, `failed to lstat`, etc.). Symlink detection yields an explicit \"security violation\" error rather than a generic I/O error.\n- `WriteFileSafely`: all failure paths return wrapped errors (directory creation, temp write, sync, close, chmod, rename). No panic usage; no sentinel values.\n\n**Deferred cleanup:** Both functions defer `f.Close()` for reads and use deferred `os.Remove` in `WriteFileSafely` to clean up temp files if the rename fails.\n\n### Gitignore Handling\n\n- **`LoadGitignore(repoRoot)`**: Opens `.gitignore` under `repoRoot`. If absent (`os.IsNotExist`), returns `nil, nil` β€” no error raised. Any other open/read failure is returned unwrapped.\n- **`ShouldIgnoreFile(repoRoot, relPath, gitIgnore)`**: Applies layered rules:\n 1. Check against compiled `GitIgnore` patterns (user-supplied + `.gitignore`).\n 2. Reject any path component starting with `.` or ending with `.egg-info`, regardless of other rules.\n 3. If extension matches a known text-source language list, accept as code.\n 4. For unknown extensions: resolve absolute path and scan first 1024 bytes for null bytes; binaries return `true`.\n\n**Side effect note:** `ShouldIgnoreFile` opens the file on every invocation to classify unknown extensions β€” this is a real I/O side effect per call, not cached.\n\n### Code Discovery Pipeline (`DiscoverCodeFiles`)\n\n**Input:** repository root + list of ignore patterns (strings).\n**Output:** slice of relative source-code paths.\n\n**Algorithmic flow:**\n1. Compile user-supplied ignore patterns and `.gitignore` entries into a single `GitIgnore` object via `LoadGitignore`.\n2. Walk the entire tree recursively using `filepath.WalkDir`. For each entry:\n - Directory: if name starts with `.` or matches any compiled pattern, skip subtree entirely.\n - File: delegate to `ShouldIgnoreFile`.\n3. Collect passing files into result slice.\n\n**Walk error handling:** If a walk callback returns an error for a single entry, the error is printed to `stderr` and that item is skipped (`return nil`). The overall `err` from `WalkDir` propagates at the end β€” terminal fatal errors are returned; non-fatal entries are swallowed silently.\n\n### Binary Classification (`IsBinaryFile`)\n\nScans first 1024 bytes for null bytes to distinguish text source files from binary content. Returns `true` (ignored) on any I/O error other than EOF β€” this \"fail-closed\" assumption treats unreadable files as likely binary rather than propagating the actual error.\n\n**Constants:** `binaryDetectionBufSize` is lowercase and package-private; not exported.\n\n---\n\n## Git Tools (`git_tools.go`)\n\n### External Command Execution (`RunGit`)\n\nSpawns `git --no-pager` via `exec.Command` with any additional arguments from the caller. The subprocess runs within `repoRoot` as its working directory. Stdout/stderr are captured into in-memory `bytes.Buffer`; no file, network, or DB I/O occurs.\n\n**Failure handling:** On error, wraps the original with stderr content:\n```\nfmt.Errorf(\"git command failed: %v, stderr: %s\", err, trimmedErr)\n```\nReturns `(string, error)` β€” partial stdout is still delivered alongside the error. No panics.\n\n### Repository Validation (`VerifyGitRepo`)\n\nDelegates to `RunGit` with subcommand `rev-parse --is-inside-work-tree`. If this invocation results in an error, returns a fixed message: `\"not a git repository (or any of the parent directories)\"`. The original stderr is discarded at this wrapper layer β€” normalized to a single sentinel-style message for downstream callers.\n\n---\n\n## Concurrency \u0026 State Analysis\n\n**Mutable state:** None detected across both files. All constants are `const` (compile-time); all variables (`safePath`, `dir`, `tmpFile`, `tmpName`, `patterns`, `files`, `buf`, `cmd`, `stdout`, `stderr`, etc.) are function-local and never escape their defining scope. No package-level global state is modified.\n\n**Concurrency mechanisms:** None detected. No `sync.Mutex`, channels, `atomic` types, or other synchronization primitives. Shared mutable state does not exist to protect.\n\n---\n\n## Error Handling Summary\n\n| Pattern | Where |\n|---|---|\n| Wrap-and-return (preserve stderr) | `RunGit` |\n| Normalize to sentinel message | `VerifyGitRepo` |\n| Sentinel + wrap with context | `ReadFileSafely`, `WriteFileSafely`, `LoadGitignore` |\n| Boolean return, resolve failures as \"ignored\" | `ShouldIgnoreFile` |\n| Swallow per-entry walk errors, propagate terminal | `DiscoverCodeFiles` |\n| Fail-closed (I/O error β†’ binary assumption) | `IsBinaryFile` |\n\n**No panics** observed across either file. All errors are returned to callers as Go values." + } +} \ No newline at end of file diff --git a/wiki/architecture.md b/wiki/architecture.md new file mode 100644 index 0000000..a0aefb5 --- /dev/null +++ b/wiki/architecture.md @@ -0,0 +1,245 @@ +# Global Architecture Overview + +## System Boundaries & Data Flow + +Code Reducer is structured as four independent subsystems with no shared mutable state: **config**, **engine**, **security**, and **tools**. Each module receives caller-provided arguments, operates on local state only, and returns resultsβ€”no synchronization primitives cross boundaries. + +**Data flow**: `internal/config` resolves a fully-populated configuration β†’ `internal/engine` consumes that config to drive an LLM pipeline β†’ `internal/security` enforces path containment for all filesystem operations the engine performs β†’ `internal/tools` provides file I/O and git discovery primitives used by both security and engine. + +## Entry Point + +The binary entry point (`main.go`) is a thin wrapper: read error from Execute() β†’ print to stderr β†’ exit with code 1. The `cmd` module defines the root cobra command, registers subcommands (init, update, setup), resolves configuration from merged sources, routes engine events to stdout/stderr, handles OS signals for graceful shutdown, and delegates all substantive work to an unshown executeCommand entry point. + +**Signal handling**: A handler captures INT and TERM via signal.NotifyContext. A deferred call ensures context cancellation when a signal arrives. The engine runner's returned error is wrapped as `"documentation run failed: …"`. + +## Configuration Lifecycle + +`config.go` declares three exported types: `ExtractionStep`, `Config`, and compile-time constants for environment variable keys, Ollama defaults (`OllamaDefaultBaseURL = "http://localhost:11434"`, `OllamaDefaultModelID = "ornith:9b"`), file-system defaults (`DefaultDocsDir = "wiki"`, `ConfigFileName = ".code-reducer.yaml"`), and multiline prompt templates. + +`io.go` provides three exported functions: `ConfigExists(cwd string) bool`, `LoadConfig(cwd string) (*Config, error)`, and `SaveConfig(cwd string, cfg *Config) error`. + +**Atomic Write Pattern for SaveConfig**: The save operation does not write directly to the target path. Sequence: marshal β†’ apply formatting normalization (collapsing single blank lines before known section headers: `system_prompt`, `module_synthesis_prompt`, `architecture_prompt`, `file_fact_consolidation_prompt`, `extraction_steps`, `ignore`) β†’ create temp file matching `.tmp.*` in same directory β†’ write normalized YAML β†’ sync and close β†’ set permissions via `os.Chmod` β†’ atomically rename temp over original. A deferred function calls `tmpFile.Close()` and `os.Remove(tmpName)`. On success, `Close` runs twice on the temp fileβ€”Go allows this safelyβ€”and the defer's `Remove` returns an error that is silently swallowed because the original config now holds the renamed data. + +**Load-Save Roundtrip Integrity**: Loading preserves whatever formatting was originally written; normalization is applied only on write, not read. This decouples the in-memory representation from on-disk formatting expectations. `LoadConfig` returns a fresh `*Config` via unmarshaling with no shared state. Errors from `os.ReadFile` propagate directly; YAML parse failures are wrapped with `"failed to parse yaml config"` prefix chained via `%w`; marshal failures similarly wrap a descriptive message. + +**Multi-Source Configuration Resolution**: `resolve.go` exposes a single exported function: `ResolveConfig(repoRoot, modelIDFlag, numCtxFlag string) (*config.Config, error)`. Internal helper `mergeAndDeduplicate` is lowercase and not part of the public surface. Only usage (not definition) of package-level types (`Config`, `OllamaDefaultModelID`, `CodeReducerModelIDEnvKey`, `DefaultExtractionSteps`) is visible here. + +**Resolution Priority System**: Each configurable field follows its own independent priority chain, lowest to highest: **defaults β†’ YAML config file β†’ environment variables β†’ CLI flags**. The merge-and-deduplicate logic operates on local function scopes with no cross-goroutine visibility and no shared mutable state. + +**Field-Specific Priority Chains**: `ModelID` applies Default > YAML > Env Var > CLI Flag. `OllamaBaseURL` applies Default > YAML > Env Var. `OllamaNumCtx` (context size) applies Default > YAML > Env Var > CLI Flag. `DocsDir` applies Default > YAML. System prompts (`SystemPrompt`, `ModuleSynthesisPrompt`, `ArchitecturePrompt`, `FileFactConsolidationPrompt`) all start with default, then override with YAML if non-empty. + +**Validation Rules**: Numeric fields: CLI flag and env var values are parsed as integers; a value must be greater than zero to take effect. Invalid or non-positive values fall through silently. String fields: empty string is treated as "not set" β€” the lower-priority default remains active. Config file absence (config.yml): not an error; if missing, an empty `Config{}` struct is created and defaults are applied. Other parse errors are surfaced. + +**Algorithmic Flow for ResolveConfig**: (1) Load YAML config β€” if it exists and parses successfully; otherwise start with an empty struct (missing-file case). (2) Resolve extraction steps β€” use YAML value if present, else fall back to `DefaultExtractionSteps`. (3) Deduplicate ignore list from the loaded config using a map-based seen-set while preserving first-seen order. (4) For each field, apply its specific priority chain: start with built-in default constant β†’ override with YAML (if non-empty) β†’ override with environment variable (if set and valid for numeric types) β†’ override with CLI flag (highest priority; numeric types require successful integer parse and positive value). (5) Return the fully resolved `Config` struct, or an error if the YAML file fails to parse (excluding missing-file case). + +**Error Handling in ResolveConfig**: When `os.IsNotExist(err)` is true during config loading, defaults apply with no error returned. Any other error (parse failure, permission denied) is wrapped with a prefix and returned as `(*Config, nil, error)` β€” the config pointer is `nil`. Silent branches: string comparisons (`!= ""`, `> 0`) short-circuit without errors. `strconv.Atoi` failures are caught inline; env/flag values simply not used. Map lookups for deduplication have no error path. + +**Always-On Return**: The function always returns a non-nil tuple `(*Config, error)` β€” either a populated config with `nil` error, or a `nil` config with an error string. No panics anywhere in this file. + +## Engine Pipeline Architecture + +### Constants & Types + +`constants.go` declares compile-time constants: `defaultHTTPTimeout` (10 min), `maxErrorBodyBytes` (1024 bytes), `defaultChunkOverlap` (800 chars), `minNumCtxFloor` (512 tokens), `contextWindowAllocRatio` (0.75), `maxCharsMultiplier` (3), `metadataFileName` (`.metadata.json`), `agentsFileName` (`AGENTS.md`), `defaultDirPerm` (0755). Type alias `LogEventFunc` carries signature `func(EventType, string)` with no business rules defined here. + +### LLM Client Communication + +Unexported except `Message` struct (`Role string`, `Content string`). The package-private `llmClient` carries: model ID, base URL, context count, HTTP client pointer. All set once during construction; never modified afterward within this file. No mutexes or channels used. + +**Algorithmic Flow**: (1) Prepare payload: assemble request body by prepending `systemPrompt` as a system-role message before any user-supplied messages, serialize as JSON to Ollama `/api/chat`. (2) Send HTTP request: single synchronous POST with fixed timeout; no retries (fail-fast policy). (3) Handle response: on 200 OK β†’ deserialize body β†’ extract `message.content` field. Non-2xx status code β†’ error string containing both HTTP status and raw response payload. (4) Return result: extracted text content or wrapped error describing what went wrong. + +**Error Handling Patterns**: Marshal failure wraps via `%w` for caller inspection via `errors.Is`. HTTP request construction returns unwrapped; context lost for callers. Network/transport error returns unwrapped; timeout vs DNS vs connect indistinguishable to caller. Success-path body read returns unwrapped (rare). JSON unmarshal failure wraps via `%w`. Non-OK status response uses custom message discarding read error (`_`); uses undefined `maxErrorBodyBytes` constant. No retry logic. + +### Runner & Entry Point + +`Runner` wraps a config pointer and exposes a single `Run(ctx, repoRoot, mode, onEvent)` method that is the only public entry point into the pipeline. It performs four sequential operations: (1) Gitignore lockfile maintenance β€” attempts to append the lockfile path to `.gitignore`. Failure is logged via `onEvent(EventStatus, ...)` and execution continues; no error returned. (2) Repository lock acquisition β€” calls `security.AcquireLock(repoRoot)`. On failure, returns wrapped error: `"failed to acquire repository lock: %w"`. No partial work proceeds without the lock. (3) Mode dispatch β€” constructs an LLM client from config fields (`cfg.ModelID`, `cfg.OllamaBaseURL`, `cfg.OllamaNumCtx`) and instantiates an `orchestrator`. Dispatches to either `RunInit` or `RunUpdate`; any other mode returns `"unsupported mode: %s"`. (4) Deferred lock release β€” `defer lock.Unlock()` runs after the pipeline completes, regardless of success or failure. + +The `Runner.cfg` field is set once in `NewRunner(cfg)` and never mutated within this file. The pointer type allows external reassignment but no such mutation occurs here. + +### Orchestrator & Pipeline Modes + +Unexported struct with two exported constants on the `EventType` type alias: `EventStatus = "status"` and `EventError = "error"`. The unexported `orchestrator` struct carries a method receiver; its public surface is empty. + +**RunInit β€” Cold Start Flow**: (1) setupPipeline() β†’ discover files, compute hashes, load .gitignore (swallowed), load cache (swallowed). (2) Mark every directory as affected (full regeneration). (3) Synthesize root summary from full tree via LLM calls. (4) Generate architecture.md + quickstart.md using root summary. (5) Write agent guidelines file (append if marker missing, overwrite otherwise). (6) Persist cache to disk β†’ complete. + +**RunUpdate β€” Incremental Refresh Flow**: (1) setupPipeline() β†’ discover current files, compute hashes. (2) If extraction steps changed β†’ invalidate entire cache (.StepsHash mismatch resets Files and Modules maps). (3) Classify all discovered files into Added/Modified/Deleted buckets by comparing current vs cached SHA-256 hashes. (4) Build directory tree from allowed (existing + new) files only. (5) Purge stale module summaries: for any `cache.Modules` path not represented in the live tree, delete both the cache entry and the physical `.md` file (`_ = os.Remove(...)` β€” error swallowed). (6) Compute affected directories; propagate upward through parent dirs. (7) If `len(affectedDirs) == 0` β†’ return early (no-op, docs are up to date). (8) Synthesize root summary from the affected region via LLM calls. (9) Re-generate architecture.md + quickstart.md only if `.` is in `affectedDirs`, or either file does not exist on disk, or resolution fails (treated as "not exists"). (10) Persist cache β†’ complete. + +**Pipeline Context**: Unexported struct carrying: LLM client pointer, repo root string, config pointer, `*MetadataCache` pointer, affected directory set map, precomputed file hashes map, and event logger callback. Constructed fresh per invocation; no shared instances exist across goroutines in this file. + +### Cache Persistence Layer + +Two structs with no methods: **`FileCacheEntry`** β€” pairs a SHA256 digest string (`sha256`) with extracted facts string (`facts`). Stored in `MetadataCache.Files map[string]FileCacheEntry`. **`MetadataCache`** β€” carries version int, steps hash string, files map, and modules map (module dir β†’ synthesized summary). + +**IsInitialized(repoRoot, docsDir) bool**: Returns true only if the cache file exists AND is readable in one call. Reuses the existence check from `loadMetadataCache`. No error return. + +**saveMetadataCache(repoRoot, docsDir, cache) error**: Serializes to `{docsDir}/{metadataFileName}` with indented JSON (`json.MarshalIndent(cache, "", " ")`). Marshaling errors propagate directly; write errors pass through `tools.WriteFileSafely`. The version field is pinned to the current value on every save. + +**Version Compatibility Guard**: On load: if `cache.Version != currentCacheVersion` (currently 1), returns a fresh zero-valued `MetadataCache{}` with nil error. No warning, no panicβ€”caller sees success with an empty cache. This prevents stale metadata from corrupting future runs without explicit migration handling. + +**Fault-Tolerant Initialization**: If the file does not exist (`os.ErrNotExist` sentinel matched via `errors.Is`), returns a populated zero-valued cache with nil error. If it exists but cannot be read or unmarshaled, returns wrapped error `"failed to read metadata cache: %w"`. Callers can distinguish "not yet initialized" from "corrupt data." + +### Chunking & Reduction Engine + +All unexported; package-private. Implements a **context-aware recursive reduction** for LLM-based synthesis: (1) Split: Input list divided into batches whose combined character count fits within `contextSize Γ— maxCharsMultiplier`. (2) Reduce each batch: Each batch sent to LLM via domain-specific prompt β†’ one summary string per batch. If only one batch exists, returned directly; otherwise summaries concatenated and fed back into step 1 recursively until a single result remains. (3) Graceful truncation: Items exceeding `maxChars` are truncated with `"...\n...[truncated]"`. If every item exceeds the per-item budget (`allowedPerItem`), all items uniformly truncated to avoid infinite loops. (4) Context cancellation: Checked at entry points and before each recursive reduction round; cancellation errors propagate as-is. + +**chunkTextWithOverlap**: Text-splitting utility with overlap between adjacent chunks. Appears available for reuse elsewhere but not referenced by any function here. + +### Synthesis Core + +All types and functions unexported (`pipelineContext`, `synthesizeNode`). Package-private. + +**Recursive Directory Synthesis**: A directory's summary is derived by recursively synthesizing children (subdirectories) and files together. Final result stored in cache under `cache.Modules[node.Path]` and written to disk at `docs/modules/`. + +**Caching Strategy with Hash-Based Invalidation**: Module-level: Stores synthesized summary per directory path. Reused when affected status indicates no changes (short-circuit return). File-level: Stores SHA256 hash + extracted facts in `cache.Files[f]`. Cache hit determined by comparing precomputed or read file hash against cached entry; only re-reads on miss. + +**Multi-Step Extraction Pipeline Per File Chunk**: (1) System prompt augmented with current extraction step's prompt. (2) Single LLM call extracts facts from chunk β†’ one fact per step. (3) All steps produce respective facts for the same file β†’ consolidated via `reduceFileFacts`. + +**Synthesis Order & Assembly**: (1) File facts (sorted child name order). (2) Child subdirectory summaries (after all files processed). (3) Combined list reduced to final summary via `reduceInChunks` at directory level. + +**Affected-Dir Pruning**: If a directory path has no affected status AND a cached module summary exists β†’ short-circuit return cached value without reprocessing children or files. + +### Tree & Change Detection + +Exported types: **`FileChange`** β€” `Path string`, `Status string` ("Added", "Modified", "Deleted"). **`DirNode`** β€” `Path string`, `Files []string`, `Children map[string]*DirNode`. + +**Three-Phase Algorithm**: (1) Build tree: Given raw file paths, split each on `/`, create intermediate `DirNode` entries for subdirectories, attach files to leaf nodes via `buildTree`. (2) Determine affected directories from changes: For each input change, mark directly changed file as affected (if exists in current set). When a file is Deleted, mark parent directory as affected. Walk recursively and check additional conditions: if node path matches `docs/modules/` but no such doc exists on disk β†’ affected; if node has empty cached metadata entry β†’ affected. (3) Propagate status upward: Recursive walk of `DirNode` tree propagates any `affected` flag from children to parents so a parent is considered affected if any child is. + +### JSON Parser Utility + +**stripOuterMarkdownFence(input) string**: Single-pass parser that strips markdown or JSON code fences from strings: (1) Trims whitespace from input. (2) Attempts regex match against pattern capturing triple backticks with 3+ chars, optionally followed by `markdown` or `json`, then all content between opening and closing fences (group 1). (3) If matched β†’ returns trimmed captured inner content. (4) If no match β†’ returns original trimmed input unchanged. No error return parameter. `regexp.MustCompile` panics on invalid pattern at package initβ€”unhandled within this file, propagates to any caller importing the package. + +### Utility Functions for Filename Generation and Event Logging + +This file provides two unexported utility functions supporting the engine's documentation generation pipeline and optional event logging instrumentation. Neither function communicates with external systems, modifies shared state, or returns errors. All operations are pure in-memory transformations. + +**toSafeMarkdownFilename β€” Module Path to Markdown Filename Conversion**: Transforms a module path into a safe markdown documentation filename, enforcing the engine's implicit one-to-one mapping contract between modules and their `.md` documentation files. The root module is special-cased to default to `root.md`. Data flow: (1) Input: a raw string representing a module path (e.g., `"internal/engine"`). (2) Character substitution: every `/` character in the input is replaced with `_`. (3) Extension appended: `.md` is concatenated to produce the final filename. (4) Output: a single `string` containing the resulting markdown filename. No filesystem, network, database, or other external I/O occurs during execution. The function returns only a result string; no error type is returned. Invalid input characters (anything other than `/`) are passed through unmodified β€” callers receive no escape hatch for detecting malformed paths. Pure: no mutable state, no package-level globals modified, no struct fields touched. + +**makeLogEvent β€” Optional Event Callback Adapter**: Wires an optional callback into a logging pipeline. When invoked with an event type and message pair, it either forwards the call to the registered handler or silently discards it. This makes the logging subsystem purely additive β€” no required consumer exists for the engine's log events; instrumentation is opt-in only. Data flow: (1) Input: a user-provided `onEvent` callback of type `LogEventFunc`. (2) Closure construction: returns a new function accepting `(EventType, message)` pairs. (3) Invocation behavior: if `onEvent == nil`, the returned closure does nothing when called β€” no panic, no error, no log output. If `onEvent` is non-nil, the call is forwarded to it with the provided type and message arguments. No I/O anywhere in this flow. Pure: no mutable state, no package-level globals modified. Silent failure path: when `onEvent == nil`, events are swallowed without notice or error wrapping. The returned function is always valid (non-nil), so callers can invoke it safely regardless of whether a handler was registered. + +**Referenced Types**: The following types are referenced in this file but not defined within it: `EventType` β€” categorizes log events into discrete types; carried as the first argument to event handlers and consumed by the returned closure from `makeLogEvent`. `LogEventFunc` β€” function signature for user-provided event handlers. Expected to accept `(EventType, message string)` or equivalent parameters. `Event` (struct) β€” domain model with at least two fields: `Type` (`EventType`) and `Message` (`string`). Events are tracked as discrete, typed records carrying descriptive payloads. + +**Concurrency & State Summary**: Mutable state: None. Both functions operate entirely on their inputs and return values without touching package-level variables or struct fields. Concurrency mechanisms: None. No mutexes, channels, atomic operations, or synchronization primitives are used in this file. + +## Security Invariants + +### Module Responsibility + +This module enforces two non-negotiable invariants for any code-reducer session: (1) the working directory must remain strictly within the repository root boundary, and (2) only one process may hold an exclusive lock on the shared resource at any given moment. It also maintains a `.gitignore` entry that excludes the lockfile from version control while ensuring it is automatically tracked if absent. + +The module communicates with the outside world indirectly: `security.go` performs filesystem I/O and returns sentinel error values to callers; those sentinels are defined in `errors.go`. No panic recovery, no error wrapping beyond `%w`, no silent swallowing of non-trivial failures occurs within these files. + +### Sentinel Error Contract + +Package-level variables: **`ErrPathTraversal`** (`error`) β€” returned when a resolved path escapes outside the repository root boundary. **`ErrLockHeld`** (`error`) β€” returned when another code-reducer process holds an exclusive file lock on the shared resource. Both variables are initialized once via `errors.New` and remain read-only throughout their scope. No mutable state, no concurrency primitives, no panic handling exists in this file. Callers detect violations by matching against these sentinels using `errors.Is(err, ErrPathTraversal)` or equivalent patterns; because the errors are unwrapped as-is (no wrapping), identity is preserved directly through the error chain. + +**Domain Concepts**: Path Traversal Violation (`ErrPathTraversal`) β€” signals when a resolved path escapes outside the repository root boundary. This is a classic path traversal / directory escape guardrail. Lock Conflict (`ErrLockHeld`) β€” signals when another process already holds an exclusive file lock, indicating contention on a shared resource. + +### Lock Acquisition and Release + +**Struct Definition**: `SimpleLock` carries: `lockPath string` (absolute path to the lockfile on disk), `file *os.File` (open file descriptor for the lockfile, or nil if not acquired), `mu sync.Mutex` (protects internal struct fields during `Unlock()`), `closed bool` (marks whether the lock has been released and the file descriptor closed). + +**`*SimpleLock.Unlock()`**: Closes the lock file descriptor and removes the lockfile in a thread-safe manner. **Idempotency contract**: Calling `Unlock()` on an already-closed lock returns `nil`. This is intentional, not accidental. Error handling inside the method: If `l.closed == true` β†’ return `nil`. Close the file descriptor (`os.File.Close`) β€” error captured in local variable; if close fails, set `file = nil` so a subsequent close becomes a no-op. Remove the lockfile via `os.Remove`. If both close and remove fail: remove error overwrites prior close error only when close returned without error. If close failed first, any remove error is ignored. This means the final returned error reflects whichever operation last produced an error, not necessarily the most severe one β€” an intentional design choice worth flagging for callers. If `os.IsNotExist(err)` on remove β†’ silently accepted as normal idempotent cleanup. + +### Path Resolution + +**Signature**: `func SafeResolve(repoRoot, inputPath string) (string, error)`. **Responsibility**: Resolve any input path strictly inside the repository root. The function performs a multi-step containment check to defeat symlink-based escape vectors and physical directory traversal. + +**Algorithmic Steps**: (1) Call `filepath.Abs` on `inputPath`. (2) Evaluate symlinks on the existing ancestor via `EvalSymlinks` β€” prevents symlink-based escape from parent directories that exist but are symbolic links. (3) Walk up from the target until a physically-existing directory is found, then re-evaluate symlinks on that ancestor via `EvalSymlinks` again. (4) Rejoin with the remaining suffix components after the physical ancestor boundary. (5) Verify final result is under `resolvedRoot` via relative path check (`filepath.Rel`). + +**Error Handling**: Any error from `Abs`, first `EvalSymlinks`, second `EvalSymlinks`, or `Lstat` (non-exist) wraps with descriptive message using `fmt.Errorf(..., err)` β€” underlying cause recovered via `%w`. Unrecognized error from `Lstat` (`!os.IsNotExist`) returns immediately as wrapped error. Path escapes repo root (`..` prefix or non-nil rel) returns sentinel-style error wrapping `ErrPathTraversal` with the original input path. + +### Lock Acquisition + +**Signature**: `func AcquireLock(repoRoot string) (*SimpleLock, error)`. **Responsibility**: Establish an exclusive process-level lock on a shared resource. Uses `O_EXCL` flag for atomic creation of the lockfile β€” if the lockfile already exists, acquisition fails immediately without silent success. + +**Lockfile Lifecycle**: Resolve path calls `SafeResolve(repoRoot)` to obtain absolute lock path. Create atomically opens file with `O_WRONLY | O_CREATE | O_EXCL` β€” atomic exclusive creation prevents race between existence check and open. Record identity writes PID of the acquiring process to the newly created lockfile. Failure cleanup on failure: closes file, removes lockfile. + +**Error Handling**: `SafeResolve` fails propagates as-is to caller. File open fails AND error is `os.IsExist` (lock already held) returns wrapped sentinel error wrapping `ErrLockHeld`, including the lock path β€” caller can detect stale-lock condition specifically via `errors.Is(err, ErrLockHeld)`. File open fails with any other error wraps: `"failed to acquire lock at %s: "`. Write PID fails file closed, lockfile removed (cleanup), wrapped error returned. + +### Gitignore Maintenance + +**Signature**: `func EnsureGitignoreHasLockfile(repoRoot string) (error)`. **Responsibility**: Maintain the convention that the lockfile is excluded from version control but automatically tracked if missing. Appends the lockfile entry to `.gitignore` only if not already present; uses atomic rename via temp file + sync before close to prevent partial writes. + +**Algorithmic Steps**: (1) Resolve `.gitignore` path via `SafeResolve(repoRoot)`. (2) Read existing `.gitignore` content (`os.ReadFile`). (3) Parse lines, check if lockfile entry already exists. (4) If not present: construct new content with the lockfile entry appended (with trailing-newline normalization). (5) Create a temp file in same directory as `.gitignore`. (6) Write to temp file β†’ call `Sync()` on it β†’ close it β†’ `Chmod` it. (7) Rename temp file over original `.gitignore`. + +**Error Handling**: `SafeResolve` fails propagates as-is. Read fails AND not `os.IsNotExist` (file exists but read error) wraps: `"error reading .gitignore: "` β€” caller must handle missing-file vs I/O-error distinction separately. CreateTemp fails wraps with descriptive message including `%w`. Write to temp file fails wraps; temp file is NOT cleaned up (defer handles close + remove, but write error returns before cleanup path executes). Sync fails wraps β€” note that `os.Chmod` and subsequent rename may have already run on a partially-written temp file if this error occurs late in the sequence. Close fails wraps. Chmod fails wraps. Rename (atomic swap) fails wraps with descriptive message. + +The deferred function (`tmpFile.Close()` + `os.Remove(tmpName)`) runs regardless of return path. Because the function returns before the defer has a chance to execute in some failure paths, cleanup may not happen for intermediate states β€” but since we're using temp file with unique name and same-directory rename, the OS-level temp file is cleaned by `os.Remove` in the defer on any error return path. If *write* fails mid-stream before close/rename, the deferred Remove will clean up the partially-written temp file. + +### Data Flow Summary + +``` +[Initialize] β†’ [Ensure path safety for all repo-root operations (SafeResolve)] β†’ +[If no process holds the lock] β†’ [Create lockfile with PID via O_EXCL (AcquireLock)] β†’ +[Run business logic within locked session] β†’ +[Append entry to .gitignore if needed (EnsureGitignoreHasLockfile)] +``` + +The two non-obvious constraints are: (1) path resolution walks symlinked ancestors before rejoining suffix components, and (2) gitignore updates use atomic rename with explicit sync before close. + +**Key Patterns Observed Across the Module**: Error wrapping throughout β€” All functions use `fmt.Errorf(..., err)` with `%w`, making root-cause recovery possible via `errors.Is()`. Custom sentinel errors β€” At least two custom error variables referenced: `ErrPathTraversal` and `ErrLockHeld`. These enable typed error detection by callers without requiring unwrapping logic on this file's side. No panic anywhere β€” All failure paths return to caller with an error value. Idempotent operations deliberately silent β€” `Unlock()` on already-closed lock returns nil; missing `.gitignore` in EnsureGitignoreHasLockfile returns nil. These are design choices, not accidentals. Atomic rename pattern for .gitignore β€” Write to temp β†’ Sync β†’ Close β†’ Chmod β†’ Rename is the standard atomic-update pattern. The `Sync()` call before close is explicit and unusual (normally deferred), but here it's called eagerly before close in this implementation. No global package-level mutable state exists in either file β€” all state is either local to a function or encapsulated within the `SimpleLock` receiver, which is accessed only through its methods. + +## Tools Subsystem + +### Module Responsibility + +The `internal/tools` package provides repository-scanning primitives for a codebase-analysis tool. It exposes two functional domains: **safe local file I/O** (`file_tools.go`) and **git subprocess orchestration** (`git_tools.go`). Together they enable the caller to discover source-code files, classify their content type, read/write artifacts safely, and validate that the target directory is a git work-tree before proceeding. + +Data flow follows a two-phase pattern: (1) `VerifyGitRepo` (or an equivalent pre-flight check) confirms the working directory corresponds to a valid git repository by invoking `git rev-parse --is-inside-work-tree`. (2) Once validated, `DiscoverCodeFiles` walks the entire tree under that root, applying layered ignore rules and returning only text-source files. Both domains share these architectural constraints: **no panics**, **error wrapping** (stderr is preserved for diagnostics), and **no network/DB calls**. All I/O operates within a single local process or on the local filesystem. + +### File Operations + +**Safe Read / Write Primitive**: `ReadFileSafely` and `WriteFileSafely` implement TOCTOU-safe file operations using the temp-file + rename pattern. Shared sequence: (1) Resolve a virtual path to an absolute filesystem path via `SafeResolve`. (2) On write: create parent directories with `defaultDirPerm`; on read: no directory creation. (3) Open or create a temporary file in the target directory using `os.CreateTemp`. (4) Write content (for writes) or read from the resolved path (for reads). (5) Close and sync (writes only); close (reads only). (6) Rename temp into final destination for writes; return contents for reads. + +**Symlink detection**: Both paths verify the target is not a symlink via `os.Lstat`/`f.Stat`. Pre-open checks prevent TOCTOU swap attacks where an attacker replaces a real file with a symlink between resolution and open. Post-open checks catch any state changes after the initial open. + +**Error semantics**: `ReadFileSafely`: returns `nil, err` on failure; wraps each step with context strings (`failed to open file`, `failed to lstat`, etc.). Symlink detection yields an explicit "security violation" error rather than a generic I/O error. `WriteFileSafely`: all failure paths return wrapped errors (directory creation, temp write, sync, close, chmod, rename). No panic usage; no sentinel values. Deferred cleanup: Both functions defer `f.Close()` for reads and use deferred `os.Remove` in `WriteFileSafely` to clean up temp files if the rename fails. + +**Gitignore Handling**: **`LoadGitignore(repoRoot)`**: Opens `.gitignore` under `repoRoot`. If absent (`os.IsNotExist`), returns `nil, nil` β€” no error raised. Any other open/read failure is returned unwrapped. **`ShouldIgnoreFile(repoRoot, relPath, gitIgnore)`**: Applies layered rules: (1) Check against compiled `GitIgnore` patterns (user-supplied + `.gitignore`). (2) Reject any path component starting with `.` or ending with `.egg-info`, regardless of other rules. (3) If extension matches a known text-source language list, accept as code. (4) For unknown extensions: resolve absolute path and scan first 1024 bytes for null bytes; binaries return `true`. + +**Side effect note**: `ShouldIgnoreFile` opens the file on every invocation to classify unknown extensions β€” this is a real I/O side effect per call, not cached. + +### Code Discovery Pipeline + +**Input**: repository root + list of ignore patterns (strings). **Output**: slice of relative source-code paths. + +**Algorithmic flow**: (1) Compile user-supplied ignore patterns and `.gitignore` entries into a single `GitIgnore` object via `LoadGitignore`. (2) Walk the entire tree recursively using `filepath.WalkDir`. For each entry: Directory: if name starts with `.` or matches any compiled pattern, skip subtree entirely. File: delegate to `ShouldIgnoreFile`. (3) Collect passing files into result slice. + +**Walk error handling**: If a walk callback returns an error for a single entry, the error is printed to `stderr` and that item is skipped (`return nil`). The overall `err` from `WalkDir` propagates at the end β€” terminal fatal errors are returned; non-fatal entries are swallowed silently. + +### Binary Classification + +**IsBinaryFile**: Scans first 1024 bytes for null bytes to distinguish text source files from binary content. Returns `true` (ignored) on any I/O error other than EOF β€” this "fail-closed" assumption treats unreadable files as likely binary rather than propagating the actual error. + +**Constants**: `binaryDetectionBufSize` is lowercase and package-private; not exported. + +### Git Tools + +**External Command Execution (`RunGit`)**: Spawns `git --no-pager` via `exec.Command` with any additional arguments from the caller. The subprocess runs within `repoRoot` as its working directory. Stdout/stderr are captured into in-memory `bytes.Buffer`; no file, network, or DB I/O occurs. + +**Failure handling**: On error, wraps the original with stderr content: +``` +fmt.Errorf("git command failed: %v, stderr: %s", err, trimmedErr) +``` +Returns `(string, error)` β€” partial stdout is still delivered alongside the error. No panics. + +**Repository Validation (`VerifyGitRepo`)**: Delegates to `RunGit` with subcommand `rev-parse --is-inside-work-tree`. If this invocation results in an error, returns a fixed message: `"not a git repository (or any of the parent directories)"`. The original stderr is discarded at this wrapper layer β€” normalized to a single sentinel-style message for downstream callers. + +### Concurrency & State Analysis + +**Mutable state**: None detected across both files. All constants are `const` (compile-time); all variables (`safePath`, `dir`, `tmpFile`, `tmpName`, `patterns`, `files`, `buf`, `cmd`, `stdout`, `stderr`, etc.) are function-local and never escape their defining scope. No package-level global state is modified. + +**Concurrency mechanisms**: None detected. No `sync.Mutex`, channels, `atomic` types, or other synchronization primitives. Shared mutable state does not exist to protect. + +### Error Handling Summary + +| Pattern | Where | +|---|---| +| Wrap-and-return (preserve stderr) | `RunGit` | +| Normalize to sentinel message | `VerifyGitRepo` | +| Sentinel + wrap with context | `ReadFileSafely`, `WriteFileSafely`, `LoadGitignore` | +| Boolean return, resolve failures as "ignored" | `ShouldIgnoreFile` | +| Swallow per-entry walk errors, propagate terminal | `DiscoverCodeFiles` | +| Fail-closed (I/O error β†’ binary assumption) | `IsBinaryFile` | + +**No panics** observed across either file. All errors are returned to callers as Go values. \ No newline at end of file diff --git a/wiki/modules/internal.md b/wiki/modules/internal.md new file mode 100644 index 0000000..6a08328 --- /dev/null +++ b/wiki/modules/internal.md @@ -0,0 +1,259 @@ +# Code Reducer β€” Architecture Documentation + +## Module Responsibility & Data Flow + +Code Reducer is a multi-phase LLM analysis pipeline that generates technical documentation from source code. The system operates across four subsystems: **config** (pipeline configuration and lifecycle), **engine** (recursive, context-window-aware synthesis orchestration), **security** (path containment and process mutual exclusion), and **tools** (repository scanning and safe I/O primitives). + +Data flow is sequential across subsystem boundaries: `internal/config` resolves a fully-populated `*Config` via multi-source priority merging; `internal/engine` consumes that config to drive an LLM pipeline; `internal/security` enforces path containment for all filesystem operations the engine performs; `internal/tools` provides file I/O and git discovery primitives used by both security and engine subsystems. + +No shared mutable state exists across subsystem boundaries. Each module operates on caller-provided arguments and returns results. No mutexes, channels, or atomic operations appear anywhere in the codebase. + +--- + +## Subsystem: internal/config + +### Configuration Types and Defaults + +`config.go` declares three exported types: `ExtractionStep` (per-phase identifiers carrying a name string and prompt text), `Config` (aggregate configuration container referenced by all exported functions), and compile-time constants for environment variable keys, Ollama default values (`OllamaDefaultBaseURL = "http://localhost:11434"`, `OllamaDefaultModelID = "ornith:9b"`, `OllamaDefaultNumCtx = 8192`), file-system defaults (`DefaultDocsDir = "wiki"`, `ConfigFileName = ".code-reducer.yaml"`), and multiline prompt templates. + +`DefaultExtractionSteps` is a pre-populated slice containing four named phases: `API_SIGNATURES`, `BUSINESS_LOGIC`, `STATE_AND_CONCURRENCY`, `ERRORS_AND_SIDE_EFFECTS`. Initialized at declaration only; no post-initialization mutations occur in this file. Internal helper variable `configFilePerm` (value 0600) is referenced by `io.go` but not declared here. No exported interfaces or methods exist on these types. + +### Configuration File Lifecycle + +`io.go` provides three exported functions: `ConfigExists(cwd string)` returns a boolean, `LoadConfig(cwd string)` returns `(*Config, error)`, and `SaveConfig(cwd string, cfg *Config)` returns an error. All external type references (`Config`) come from the same package. + +**Atomic Write Pattern for SaveConfig:** The save operation does not write directly to the target path. The sequence is: marshal `cfg` into YAML bytes β†’ apply formatting normalization (collapsing single blank lines to double blank lines before known section headers: `system_prompt`, `module_synthesis_prompt`, `architecture_prompt`, `file_fact_consolidation_prompt`, `extraction_steps`, `ignore`) β†’ create a temp file matching `.tmp.*` in the same directory β†’ write normalized YAML into it β†’ sync and close β†’ set permissions via `os.Chmod` β†’ atomically rename temp over original. Readers never observe a partially-written config during save because the old file remains intact until the new one is fully ready to replace it. + +**Load-Save Roundtrip Integrity:** Loading preserves whatever formatting was originally written; normalization is applied only on write, not read. This decouples the in-memory representation from on-disk formatting expectations. `LoadConfig` returns a fresh `*Config` via unmarshaling with no shared state. Errors from `os.ReadFile` propagate directly; YAML parse failures are wrapped with a `"failed to parse yaml config"` prefix chained via `%w`; marshal failures similarly wrap a descriptive message. + +**Error Propagation Strategy:** When `os.ReadFile` fails in `LoadConfig`, the OS error is returned unwrapped. Parse and write failures across all save-path operations (temp create, write, sync, close, chmod, rename) are wrapped with context via `%w`. `ConfigExists` uses an `err == nil` check instead of returning the error; no wrapping; raw error is reused as a boolean condition. A deferred function calls `tmpFile.Close()` and `os.Remove(tmpName)` in `SaveConfig`; errors from these cleanup calls are swallowed (not returned). On success, `Close` runs twice on the temp fileβ€”Go allows this safelyβ€”and the defer's `Remove` returns an error that is silently swallowed because the original config now holds the renamed data. + +### Multi-Source Configuration Resolution + +`resolve.go` exposes a single exported function: `ResolveConfig(repoRoot, modelIDFlag, numCtxFlag string)` returning `(*config.Config, error)`. Internal helper `mergeAndDeduplicate` is lowercase and not part of the public surface. Only usage (not definition) of package-level types (`Config`, `OllamaDefaultModelID`, `CodeReducerModelIDEnvKey`, `DefaultExtractionSteps`) is visible here. + +**Resolution Priority System:** Each configurable field follows its own independent priority chain, lowest to highest: **defaults β†’ YAML config file β†’ environment variables β†’ CLI flags**. The merge-and-deduplicate logic operates on local function scopes with no cross-goroutine visibility and no shared mutable state. + +**Field-Specific Priority Chains:** `ModelID` applies Default > YAML > Env Var > CLI Flag. `OllamaBaseURL` applies Default > YAML > Env Var. `OllamaNumCtx` (context size) applies Default > YAML > Env Var > CLI Flag. `DocsDir` applies Default > YAML. System prompts (`SystemPrompt`, `ModuleSynthesisPrompt`, `ArchitecturePrompt`, `FileFactConsolidationPrompt`) all start with default, then override with YAML if non-empty. + +**Validation Rules:** Numeric fields: CLI flag and env var values are parsed as integers; a value must be greater than zero to take effect. Invalid or non-positive values fall through silently. String fields: empty string is treated as "not set" β€” the lower-priority default remains active. Config file absence (`config.yml`): not an error; if missing, an empty `Config{}` struct is created and defaults are applied. Other parse errors are surfaced. + +**Algorithmic Flow for ResolveConfig:** (1) Load YAML config β€” if it exists and parses successfully; otherwise start with an empty struct (missing-file case). (2) Resolve extraction steps β€” use YAML value if present, else fall back to `DefaultExtractionSteps`. (3) Deduplicate ignore list from the loaded config using a map-based seen-set while preserving first-seen order. (4) For each field, apply its specific priority chain: start with built-in default constant β†’ override with YAML (if non-empty) β†’ override with environment variable (if set and valid for numeric types) β†’ override with CLI flag (highest priority; numeric types require successful integer parse and positive value). (5) Return the fully resolved `Config` struct, or an error if the YAML file fails to parse (excluding missing-file case). + +**Error Handling in ResolveConfig:** When `os.IsNotExist(err)` is true during config loading, defaults apply with no error returned. Any other error (parse failure, permission denied) is wrapped with a prefix and returned as `(*Config, nil, error)` β€” the config pointer is `nil`. Silent branches: string comparisons (`!= ""`, `> 0`) short-circuit without errors. `strconv.Atoi` failures are caught inline; env/flag values simply not used. Map lookups for deduplication have no error path. + +**Always-On Return:** The function always returns a non-nil tuple `(*Config, error)` β€” either a populated config with `nil` error, or a `nil` config with an error string. No panics anywhere in this file. + +--- + +## Subsystem: internal/engine + +### Constants & Types + +`constants.go` declares compile-time constants: `defaultHTTPTimeout` (10 min), `maxErrorBodyBytes` (1024 bytes), `defaultChunkOverlap` (800 chars), `minNumCtxFloor` (512 tokens), `contextWindowAllocRatio` (0.75), `maxCharsMultiplier` (3), `metadataFileName` (`.metadata.json`), `agentsFileName` (`AGENTS.md`), `defaultDirPerm` (0755). Type alias `LogEventFunc` carries signature `func(EventType, string)` with no business rules defined here. + +### LLM Client Communication + +Unexported except `Message` struct (`Role string`, `Content string`). The package-private `llmClient` carries: model ID, base URL, context count, HTTP client pointer. All set once during construction; never modified afterward within this file. No mutexes or channels used. + +**Algorithmic Flow:** (1) Prepare payload: assemble request body by prepending `systemPrompt` as a system-role message before any user-supplied messages, serialize as JSON to Ollama `/api/chat`. (2) Send HTTP request: single synchronous POST with fixed timeout; no retries (fail-fast policy). (3) Handle response: on 200 OK β†’ deserialize body β†’ extract `message.content` field. Non-2xx status code β†’ error string containing both HTTP status and raw response payload. (4) Return result: extracted text content or wrapped error describing what went wrong. + +**Error Handling Patterns:** Marshal failure wraps via `%w` for caller inspection via `errors.Is`. HTTP request construction returns unwrapped; context lost for callers. Network/transport error returns unwrapped; timeout vs DNS vs connect indistinguishable to caller. Success-path body read returns unwrapped (rare). JSON unmarshal failure wraps via `%w`. Non-OK status response uses custom message discarding read error (`_`); uses undefined `maxErrorBodyBytes` constant. No retry logic. + +### Runner & Entry Point + +`Runner` wraps a config pointer and exposes a single `Run(ctx, repoRoot, mode, onEvent)` method that is the only public entry point into the pipeline. It performs four sequential operations: (1) Gitignore lockfile maintenance β€” attempts to append the lockfile path to `.gitignore`. Failure is logged via `onEvent(EventStatus, ...)` and execution continues; no error returned. (2) Repository lock acquisition β€” calls `security.AcquireLock(repoRoot)`. On failure, returns wrapped error: `"failed to acquire repository lock: %w"`. No partial work proceeds without the lock. (3) Mode dispatch β€” constructs an LLM client from config fields (`cfg.ModelID`, `cfg.OllamaBaseURL`, `cfg.OllamaNumCtx`) and instantiates an `orchestrator`. Dispatches to either `RunInit` or `RunUpdate`; any other mode returns `"unsupported mode: %s"`. (4) Deferred lock release β€” `defer lock.Unlock()` runs after the pipeline completes, regardless of success or failure. + +The `Runner.cfg` field is set once in `NewRunner(cfg)` and never mutated within this file. The pointer type allows external reassignment but no such mutation occurs here. + +### Orchestrator & Pipeline Modes + +Unexported struct with two exported constants on the `EventType` type alias: `EventStatus = "status"` and `EventError = "error"`. The unexported `orchestrator` struct carries a method receiver; its public surface is empty. + +**RunInit β€” Cold Start Flow:** (1) setupPipeline() β†’ discover files, compute hashes, load .gitignore (swallowed), load cache (swallowed). (2) Mark every directory as affected (full regeneration). (3) Synthesize root summary from full tree via LLM calls. (4) Generate architecture.md + quickstart.md using root summary. (5) Write agent guidelines file (append if marker missing, overwrite otherwise). (6) Persist cache to disk β†’ complete. + +**RunUpdate β€” Incremental Refresh Flow:** (1) setupPipeline() β†’ discover current files, compute hashes. (2) If extraction steps changed β†’ invalidate entire cache (.StepsHash mismatch resets Files and Modules maps). (3) Classify all discovered files into Added/Modified/Deleted buckets by comparing current vs cached SHA-256 hashes. (4) Build directory tree from allowed (existing + new) files only. (5) Purge stale module summaries: for any `cache.Modules` path not represented in the live tree, delete both the cache entry and the physical `.md` file (`_ = os.Remove(...)` β€” error swallowed). (6) Compute affected directories; propagate upward through parent dirs. (7) If `len(affectedDirs) == 0` β†’ return early (no-op, docs are up to date). (8) Synthesize root summary from the affected region via LLM calls. (9) Re-generate architecture.md + quickstart.md only if `.` is in `affectedDirs`, or either file does not exist on disk, or resolution fails (treated as "not exists"). (10) Persist cache β†’ complete. + +**Pipeline Context:** Unexported struct carrying: LLM client pointer, repo root string, config pointer, `*MetadataCache` pointer, affected directory set map, precomputed file hashes map, and event logger callback. Constructed fresh per invocation; no shared instances exist across goroutines in this file. + +### Cache Persistence Layer + +Two structs with no methods: **`FileCacheEntry`** β€” pairs a SHA256 digest string (`sha256`) with extracted facts string (`facts`). Stored in `MetadataCache.Files map[string]FileCacheEntry`. **`MetadataCache`** β€” carries version int, steps hash string, files map, and modules map (module dir β†’ synthesized summary). + +**IsInitialized(repoRoot, docsDir) bool:** Returns true only if the cache file exists AND is readable in one call. Reuses the existence check from `loadMetadataCache`. No error return. + +**saveMetadataCache(repoRoot, docsDir, cache) error:** Serializes to `{docsDir}/{metadataFileName}` with indented JSON (`json.MarshalIndent(cache, "", " ")`). Marshaling errors propagate directly; write errors pass through `tools.WriteFileSafely`. The version field is pinned to the current value on every save. + +**Version Compatibility Guard:** On load: if `cache.Version != currentCacheVersion` (currently 1), returns a fresh zero-valued `MetadataCache{}` with nil error. No warning, no panicβ€”caller sees success with an empty cache. This prevents stale metadata from corrupting future runs without explicit migration handling. + +**Fault-Tolerant Initialization:** If the file does not exist (`os.ErrNotExist` sentinel matched via `errors.Is`), returns a populated zero-valued cache with nil error. If it exists but cannot be read or unmarshaled, returns wrapped error `"failed to read metadata cache: %w"`. Callers can distinguish "not yet initialized" from "corrupt data." + +### Chunking & Reduction Engine + +All unexported; package-private. Implements a **context-aware recursive reduction** for LLM-based synthesis: (1) Split: Input list divided into batches whose combined character count fits within `contextSize Γ— maxCharsMultiplier`. (2) Reduce each batch: Each batch sent to LLM via domain-specific prompt β†’ one summary string per batch. If only one batch exists, returned directly; otherwise summaries concatenated and fed back into step 1 recursively until a single result remains. (3) Graceful truncation: Items exceeding `maxChars` are truncated with `"...\n...[truncated]"`. If every item exceeds the per-item budget (`allowedPerItem`), all items uniformly truncated to avoid infinite loops. (4) Context cancellation: Checked at entry points and before each recursive reduction round; cancellation errors propagate as-is. + +**chunkTextWithOverlap:** Text-splitting utility with overlap between adjacent chunks. Appears available for reuse elsewhere but not referenced by any function here. + +### Synthesis Core + +All types and functions unexported (`pipelineContext`, `synthesizeNode`). Package-private. + +**Recursive Directory Synthesis:** A directory's summary is derived by recursively synthesizing children (subdirectories) and files together. Final result stored in cache under `cache.Modules[node.Path]` and written to disk at `docs/modules/`. + +**Caching Strategy with Hash-Based Invalidation:** Module-level: Stores synthesized summary per directory path. Reused when affected status indicates no changes (short-circuit return). File-level: Stores SHA256 hash + extracted facts in `cache.Files[f]`. Cache hit determined by comparing precomputed or read file hash against cached entry; only re-reads on miss. + +**Multi-Step Extraction Pipeline Per File Chunk:** (1) System prompt augmented with current extraction step's prompt. (2) Single LLM call extracts facts from chunk β†’ one fact per step. (3) All steps produce respective facts for the same file β†’ consolidated via `reduceFileFacts`. + +**Synthesis Order & Assembly:** (1) File facts (sorted child name order). (2) Child subdirectory summaries (after all files processed). (3) Combined list reduced to final summary via `reduceInChunks` at directory level. + +**Affected-Dir Pruning:** If a directory path has no affected status AND a cached module summary exists β†’ short-circuit return cached value without reprocessing children or files. + +### Tree & Change Detection + +Exported types: **`FileChange`** β€” `Path string`, `Status string` ("Added", "Modified", "Deleted"). **`DirNode`** β€” `Path string`, `Files []string`, `Children map[string]*DirNode`. + +**Three-Phase Algorithm:** (1) Build tree: Given raw file paths, split each on `/`, create intermediate `DirNode` entries for subdirectories, attach files to leaf nodes via `buildTree`. (2) Determine affected directories from changes: For each input change, mark directly changed file as affected (if exists in current set). When a file is Deleted, mark parent directory as affected. Walk recursively and check additional conditions: if node path matches `docs/modules/` but no such doc exists on disk β†’ affected; if node has empty cached metadata entry β†’ affected. (3) Propagate status upward: Recursive walk of `DirNode` tree propagates any `affected` flag from children to parents so a parent is considered affected if any child is. + +### JSON Parser Utility + +**stripOuterMarkdownFence(input) string:** Single-pass parser that strips markdown or JSON code fences from strings: (1) Trims whitespace from input. (2) Attempts regex match against pattern capturing triple backticks with 3+ chars, optionally followed by `markdown` or `json`, then all content between opening and closing fences (group 1). (3) If matched β†’ returns trimmed captured inner content. (4) If no match β†’ returns original trimmed input unchanged. No error return parameter. `regexp.MustCompile` panics on invalid pattern at package initβ€”unhandled within this file, propagates to any caller importing the package. + +### Utility Functions for Filename Generation and Event Logging + +This file provides two unexported utility functions supporting the engine's documentation generation pipeline and optional event logging instrumentation. Neither function communicates with external systems, modifies shared state, or returns errors. All operations are pure in-memory transformations. + +**toSafeMarkdownFilename β€” Module Path to Markdown Filename Conversion:** Transforms a module path into a safe markdown documentation filename, enforcing the engine's implicit one-to-one mapping contract between modules and their `.md` documentation files. The root module is special-cased to default to `root.md`. Data flow: (1) Input: a raw string representing a module path (e.g., `"internal/engine"`). (2) Character substitution: every `/` character in the input is replaced with `_`. (3) Extension appended: `.md` is concatenated to produce the final filename. (4) Output: a single `string` containing the resulting markdown filename. No filesystem, network, database, or other external I/O occurs during execution. The function returns only a result string; no error type is returned. Invalid input characters (anything other than `/`) are passed through unmodified β€” callers receive no escape hatch for detecting malformed paths. Pure: no mutable state, no package-level globals modified, no struct fields touched. + +**makeLogEvent β€” Optional Event Callback Adapter:** Wires an optional callback into a logging pipeline. When invoked with an event type and message pair, it either forwards the call to the registered handler or silently discards it. This makes the logging subsystem purely additive β€” no required consumer exists for the engine's log events; instrumentation is opt-in only. Data flow: (1) Input: a user-provided `onEvent` callback of type `LogEventFunc`. (2) Closure construction: returns a new function accepting `(EventType, message)` pairs. (3) Invocation behavior: if `onEvent == nil`, the returned closure does nothing when called β€” no panic, no error, no log output. If `onEvent` is non-nil, the call is forwarded to it with the provided type and message arguments. No I/O anywhere in this flow. Pure: no mutable state, no package-level globals modified. Silent failure path: when `onEvent == nil`, events are swallowed without notice or error wrapping. The returned function is always valid (non-nil), so callers can invoke it safely regardless of whether a handler was registered. + +**Referenced Types:** The following types are referenced in this file but not defined within it: `EventType` β€” categorizes log events into discrete types; carried as the first argument to event handlers and consumed by the returned closure from `makeLogEvent`. `LogEventFunc` β€” function signature for user-provided event handlers. Expected to accept `(EventType, message string)` or equivalent parameters. `Event` (struct) β€” domain model with at least two fields: `Type` (`EventType`) and `Message` (`string`). Events are tracked as discrete, typed records carrying descriptive payloads. + +**Concurrency & State Summary:** Mutable state: None. Both functions operate entirely on their inputs and return values without touching package-level variables or struct fields. Concurrency mechanisms: None. No mutexes, channels, atomic operations, or synchronization primitives are used in this file. + +--- + +## Subsystem: internal/security + +### Module Responsibility + +This module enforces two non-negotiable invariants for any code-reducer session: (1) the working directory must remain strictly within the repository root boundary, and (2) only one process may hold an exclusive lock on the shared resource at any given moment. It also maintains a `.gitignore` entry that excludes the lockfile from version control while ensuring it is automatically tracked if absent. + +The module communicates with the outside world indirectly: `security.go` performs filesystem I/O and returns sentinel error values to callers; those sentinels are defined in `errors.go`. No panic recovery, no error wrapping beyond `%w`, no silent swallowing of non-trivial failures occurs within these files. + +### Sentinel Error Contract + +Package-level variables: **`ErrPathTraversal`** (`error`) β€” returned when a resolved path escapes outside the repository root boundary. **`ErrLockHeld`** (`error`) β€” returned when another code-reducer process holds an exclusive file lock on the shared resource. Both variables are initialized once via `errors.New` and remain read-only throughout their scope. No mutable state, no concurrency primitives, no panic handling exists in this file. Callers detect violations by matching against these sentinels using `errors.Is(err, ErrPathTraversal)` or equivalent patterns; because the errors are unwrapped as-is (no wrapping), identity is preserved directly through the error chain. + +**Domain Concepts:** Path Traversal Violation (`ErrPathTraversal`) β€” signals when a resolved path escapes outside the repository root boundary. This is a classic path traversal / directory escape guardrail. Lock Conflict (`ErrLockHeld`) β€” signals when another process already holds an exclusive file lock, indicating contention on a shared resource. + +### Lock Acquisition and Release + +**Struct Definition:** `SimpleLock` carries: `lockPath string` (absolute path to the lockfile on disk), `file *os.File` (open file descriptor for the lockfile, or nil if not acquired), `mu sync.Mutex` (protects internal struct fields during `Unlock()`), `closed bool` (marks whether the lock has been released and the file descriptor closed). + +**`*SimpleLock.Unlock()`:** Closes the lock file descriptor and removes the lockfile in a thread-safe manner. **Idempotency contract:** Calling `Unlock()` on an already-closed lock returns `nil`. This is intentional, not accidental. Error handling inside the method: If `l.closed == true` β†’ return `nil`. Close the file descriptor (`os.File.Close`) β€” error captured in local variable; if close fails, set `file = nil` so a subsequent close becomes a no-op. Remove the lockfile via `os.Remove`. If both close and remove fail: remove error overwrites prior close error only when close returned without error. If close failed first, any remove error is ignored. This means the final returned error reflects whichever operation last produced an error, not necessarily the most severe one β€” an intentional design choice worth flagging for callers. If `os.IsNotExist(err)` on remove β†’ silently accepted as normal idempotent cleanup. + +### Path Resolution + +**Signature:** `func SafeResolve(repoRoot, inputPath string) (string, error)`. **Responsibility:** Resolve any input path strictly inside the repository root. The function performs a multi-step containment check to defeat symlink-based escape vectors and physical directory traversal. + +**Algorithmic Steps:** (1) Call `filepath.Abs` on `inputPath`. (2) Evaluate symlinks on the existing ancestor via `EvalSymlinks` β€” prevents symlink-based escape from parent directories that exist but are symbolic links. (3) Walk up from the target until a physically-existing directory is found, then re-evaluate symlinks on that ancestor via `EvalSymlinks` again. (4) Rejoin with the remaining suffix components after the physical ancestor boundary. (5) Verify final result is under `resolvedRoot` via relative path check (`filepath.Rel`). + +**Error Handling:** Any error from `Abs`, first `EvalSymlinks`, second `EvalSymlinks`, or `Lstat` (non-exist) wraps with descriptive message using `fmt.Errorf(..., err)` β€” underlying cause recovered via `%w`. Unrecognized error from `Lstat` (`!os.IsNotExist`) returns immediately as wrapped error. Path escapes repo root (`..` prefix or non-nil rel) returns sentinel-style error wrapping `ErrPathTraversal` with the original input path. + +### Lock Acquisition + +**Signature:** `func AcquireLock(repoRoot string) (*SimpleLock, error)`. **Responsibility:** Establish an exclusive process-level lock on a shared resource. Uses `O_EXCL` flag for atomic creation of the lockfile β€” if the lockfile already exists, acquisition fails immediately without silent success. + +**Lockfile Lifecycle:** Resolve path calls `SafeResolve(repoRoot)` to obtain absolute lock path. Create atomically opens file with `O_WRONLY | O_CREATE | O_EXCL` β€” atomic exclusive creation prevents race between existence check and open. Record identity writes PID of the acquiring process to the newly created lockfile. Failure cleanup on failure: closes file, removes lockfile. + +**Error Handling:** `SafeResolve` fails propagates as-is to caller. File open fails AND error is `os.IsExist` (lock already held) returns wrapped sentinel error wrapping `ErrLockHeld`, including the lock path β€” caller can detect stale-lock condition specifically via `errors.Is(err, ErrLockHeld)`. File open fails with any other error wraps: `"failed to acquire lock at %s: "`. Write PID fails file closed, lockfile removed (cleanup), wrapped error returned. + +### Gitignore Maintenance + +**Signature:** `func EnsureGitignoreHasLockfile(repoRoot string) (error)`. **Responsibility:** Maintain the convention that the lockfile is excluded from version control but automatically tracked if missing. Appends the lockfile entry to `.gitignore` only if not already present; uses atomic rename via temp file + sync before close to prevent partial writes. + +**Algorithmic Steps:** (1) Resolve `.gitignore` path via `SafeResolve(repoRoot)`. (2) Read existing `.gitignore` content (`os.ReadFile`). (3) Parse lines, check if lockfile entry already exists. (4) If not present: construct new content with the lockfile entry appended (with trailing-newline normalization). (5) Create a temp file in same directory as `.gitignore`. (6) Write to temp file β†’ call `Sync()` on it β†’ close it β†’ `Chmod` it. (7) Rename temp file over original `.gitignore`. + +**Error Handling:** `SafeResolve` fails propagates as-is. Read fails AND not `os.IsNotExist` (file exists but read error) wraps: `"error reading .gitignore: "` β€” caller must handle missing-file vs I/O-error distinction separately. CreateTemp fails wraps with descriptive message including `%w`. Write to temp file fails wraps; temp file is NOT cleaned up (defer handles close + remove, but write error returns before cleanup path executes). Sync fails wraps β€” note that `os.Chmod` and subsequent rename may have already run on a partially-written temp file if this error occurs late in the sequence. Close fails wraps. Chmod fails wraps. Rename (atomic swap) fails wraps with descriptive message. + +The deferred function (`tmpFile.Close()` + `os.Remove(tmpName)`) runs regardless of return path. Because the function returns before the defer has a chance to execute in some failure paths, cleanup may not happen for intermediate states β€” but since we're using temp file with unique name and same-directory rename, the OS-level temp file is cleaned by `os.Remove` in the defer on any error return path. If *write* fails mid-stream before close/rename, the deferred Remove will clean up the partially-written temp file. + +### Data Flow Summary + +``` +[Initialize] β†’ [Ensure path safety for all repo-root operations (SafeResolve)] β†’ +[If no process holds the lock] β†’ [Create lockfile with PID via O_EXCL (AcquireLock)] β†’ +[Run business logic within locked session] β†’ +[Append entry to .gitignore if needed (EnsureGitignoreHasLockfile)] +``` + +The two non-obvious constraints are: (1) path resolution walks symlinked ancestors before rejoining suffix components, and (2) gitignore updates use atomic rename with explicit sync before close. + +**Key Patterns Observed Across the Module:** Error wrapping throughout β€” All functions use `fmt.Errorf(..., err)` with `%w`, making root-cause recovery possible via `errors.Is()`. Custom sentinel errors β€” At least two custom error variables referenced: `ErrPathTraversal` and `ErrLockHeld`. These enable typed error detection by callers without requiring unwrapping logic on this file's side. No panic anywhere β€” All failure paths return to caller with an error value. Idempotent operations deliberately silent β€” `Unlock()` on already-closed lock returns nil; missing `.gitignore` in EnsureGitignoreHasLockfile returns nil. These are design choices, not accidentals. Atomic rename pattern for .gitignore β€” Write to temp β†’ Sync β†’ Close β†’ Chmod β†’ Rename is the standard atomic-update pattern. The `Sync()` call before close is explicit and unusual (normally deferred), but here it's called eagerly before close in this implementation. No global package-level mutable state exists in either file β€” all state is either local to a function or encapsulated within the `SimpleLock` receiver, which is accessed only through its methods. + +--- + +## Subsystem: internal/tools + +### Module Responsibility + +The `internal/tools` package provides repository-scanning primitives for a codebase-analysis tool. It exposes two functional domains: **safe local file I/O** (`file_tools.go`) and **git subprocess orchestration** (`git_tools.go`). Together they enable the caller to discover source-code files, classify their content type, read/write artifacts safely, and validate that the target directory is a git work-tree before proceeding. + +Data flow follows a two-phase pattern: (1) `VerifyGitRepo` (or an equivalent pre-flight check) confirms the working directory corresponds to a valid git repository by invoking `git rev-parse --is-inside-work-tree`. (2) Once validated, `DiscoverCodeFiles` walks the entire tree under that root, applying layered ignore rules and returning only text-source files. Both domains share these architectural constraints: **no panics**, **error wrapping** (stderr is preserved for diagnostics), and **no network/DB calls**. All I/O operates within a single local process or on the local filesystem. + +### File Operations + +**Safe Read / Write Primitive:** `ReadFileSafely` and `WriteFileSafely` implement TOCTOU-safe file operations using the temp-file + rename pattern. Shared sequence: (1) Resolve a virtual path to an absolute filesystem path via `SafeResolve`. (2) On write: create parent directories with `defaultDirPerm`; on read: no directory creation. (3) Open or create a temporary file in the target directory using `os.CreateTemp`. (4) Write content (for writes) or read from the resolved path (for reads). (5) Close and sync (writes only); close (reads only). (6) Rename temp into final destination for writes; return contents for reads. + +**Symlink detection:** Both paths verify the target is not a symlink via `os.Lstat`/`f.Stat`. Pre-open checks prevent TOCTOU swap attacks where an attacker replaces a real file with a symlink between resolution and open. Post-open checks catch any state changes after the initial open. + +**Error semantics:** `ReadFileSafely`: returns `nil, err` on failure; wraps each step with context strings (`failed to open file`, `failed to lstat`, etc.). Symlink detection yields an explicit "security violation" error rather than a generic I/O error. `WriteFileSafely`: all failure paths return wrapped errors (directory creation, temp write, sync, close, chmod, rename). No panic usage; no sentinel values. Deferred cleanup: Both functions defer `f.Close()` for reads and use deferred `os.Remove` in `WriteFileSafely` to clean up temp files if the rename fails. + +**Gitignore Handling:** **`LoadGitignore(repoRoot)`**: Opens `.gitignore` under `repoRoot`. If absent (`os.IsNotExist`), returns `nil, nil` β€” no error raised. Any other open/read failure is returned unwrapped. **`ShouldIgnoreFile(repoRoot, relPath, gitIgnore)`**: Applies layered rules: (1) Check against compiled `GitIgnore` patterns (user-supplied + `.gitignore`). (2) Reject any path component starting with `.` or ending with `.egg-info`, regardless of other rules. (3) If extension matches a known text-source language list, accept as code. (4) For unknown extensions: resolve absolute path and scan first 1024 bytes for null bytes; binaries return `true`. + +**Side effect note:** `ShouldIgnoreFile` opens the file on every invocation to classify unknown extensions β€” this is a real I/O side effect per call, not cached. + +### Code Discovery Pipeline + +**Input:** repository root + list of ignore patterns (strings). **Output:** slice of relative source-code paths. + +**Algorithmic flow:** (1) Compile user-supplied ignore patterns and `.gitignore` entries into a single `GitIgnore` object via `LoadGitignore`. (2) Walk the entire tree recursively using `filepath.WalkDir`. For each entry: Directory: if name starts with `.` or matches any compiled pattern, skip subtree entirely. File: delegate to `ShouldIgnoreFile`. (3) Collect passing files into result slice. + +**Walk error handling:** If a walk callback returns an error for a single entry, the error is printed to `stderr` and that item is skipped (`return nil`). The overall `err` from `WalkDir` propagates at the end β€” terminal fatal errors are returned; non-fatal entries are swallowed silently. + +### Binary Classification + +**IsBinaryFile:** Scans first 1024 bytes for null bytes to distinguish text source files from binary content. Returns `true` (ignored) on any I/O error other than EOF β€” this "fail-closed" assumption treats unreadable files as likely binary rather than propagating the actual error. + +**Constants:** `binaryDetectionBufSize` is lowercase and package-private; not exported. + +### Git Tools + +**External Command Execution (`RunGit`):** Spawns `git --no-pager` via `exec.Command` with any additional arguments from the caller. The subprocess runs within `repoRoot` as its working directory. Stdout/stderr are captured into in-memory `bytes.Buffer`; no file, network, or DB I/O occurs. + +**Failure handling:** On error, wraps the original with stderr content: +``` +fmt.Errorf("git command failed: %v, stderr: %s", err, trimmedErr) +``` +Returns `(string, error)` β€” partial stdout is still delivered alongside the error. No panics. + +**Repository Validation (`VerifyGitRepo`):** Delegates to `RunGit` with subcommand `rev-parse --is-inside-work-tree`. If this invocation results in an error, returns a fixed message: `"not a git repository (or any of the parent directories)"`. The original stderr is discarded at this wrapper layer β€” normalized to a single sentinel-style message for downstream callers. + +### Concurrency & State Analysis + +**Mutable state:** None detected across both files. All constants are `const` (compile-time); all variables (`safePath`, `dir`, `tmpFile`, `tmpName`, `patterns`, `files`, `buf`, `cmd`, `stdout`, `stderr`, etc.) are function-local and never escape their defining scope. No package-level global state is modified. + +**Concurrency mechanisms:** None detected. No `sync.Mutex`, channels, `atomic` types, or other synchronization primitives. Shared mutable state does not exist to protect. + +### Error Handling Summary + +| Pattern | Where | +|---|---| +| Wrap-and-return (preserve stderr) | `RunGit` | +| Normalize to sentinel message | `VerifyGitRepo` | +| Sentinel + wrap with context | `ReadFileSafely`, `WriteFileSafely`, `LoadGitignore` | +| Boolean return, resolve failures as "ignored" | `ShouldIgnoreFile` | +| Swallow per-entry walk errors, propagate terminal | `DiscoverCodeFiles` | +| Fail-closed (I/O error β†’ binary assumption) | `IsBinaryFile` | + +**No panics** observed across either file. All errors are returned to callers as Go values. \ No newline at end of file diff --git a/wiki/modules/internal_tools.md b/wiki/modules/internal_tools.md new file mode 100644 index 0000000..00957ae --- /dev/null +++ b/wiki/modules/internal_tools.md @@ -0,0 +1,107 @@ +# Package: internal/tools β€” Architecture Document + +## Module Responsibility + +The `internal/tools` package provides repository-scanning primitives for a codebase-analysis tool. It exposes two functional domains: **safe local file I/O** (`file_tools.go`) and **git subprocess orchestration** (`git_tools.go`). Together they enable the caller to discover source-code files, classify their content type, read/write artifacts safely, and validate that the target directory is a git work-tree before proceeding. + +Data flow follows a two-phase pattern: +1. `VerifyGitRepo` (or an equivalent pre-flight check) confirms the working directory corresponds to a valid git repository by invoking `git rev-parse --is-inside-work-tree`. +2. Once validated, `DiscoverCodeFiles` walks the entire tree under that root, applying layered ignore rules and returning only text-source files. + +Both domains share these architectural constraints: **no panics**, **error wrapping** (stderr is preserved for diagnostics), and **no network/DB calls**. All I/O operates within a single local process or on the local filesystem. + +--- + +## File Operations (`file_tools.go`) + +### Safe Read / Write Primitive + +`ReadFileSafely` and `WriteFileSafely` implement TOCTOU-safe file operations using the temp-file + rename pattern. + +**Shared sequence:** +1. Resolve a virtual path to an absolute filesystem path via `SafeResolve`. +2. On write: create parent directories with `defaultDirPerm`; on read: no directory creation. +3. Open or create a temporary file in the target directory using `os.CreateTemp`. +4. Write content (for writes) or read from the resolved path (for reads). +5. Close and sync (writes only); close (reads only). +6. Rename temp into final destination for writes; return contents for reads. + +**Symlink detection:** Both paths verify the target is not a symlink via `os.Lstat`/`f.Stat`. Pre-open checks prevent TOCTOU swap attacks where an attacker replaces a real file with a symlink between resolution and open. Post-open checks catch any state changes after the initial open. + +**Error semantics:** +- `ReadFileSafely`: returns `nil, err` on failure; wraps each step with context strings (`failed to open file`, `failed to lstat`, etc.). Symlink detection yields an explicit "security violation" error rather than a generic I/O error. +- `WriteFileSafely`: all failure paths return wrapped errors (directory creation, temp write, sync, close, chmod, rename). No panic usage; no sentinel values. + +**Deferred cleanup:** Both functions defer `f.Close()` for reads and use deferred `os.Remove` in `WriteFileSafely` to clean up temp files if the rename fails. + +### Gitignore Handling + +- **`LoadGitignore(repoRoot)`**: Opens `.gitignore` under `repoRoot`. If absent (`os.IsNotExist`), returns `nil, nil` β€” no error raised. Any other open/read failure is returned unwrapped. +- **`ShouldIgnoreFile(repoRoot, relPath, gitIgnore)`**: Applies layered rules: + 1. Check against compiled `GitIgnore` patterns (user-supplied + `.gitignore`). + 2. Reject any path component starting with `.` or ending with `.egg-info`, regardless of other rules. + 3. If extension matches a known text-source language list, accept as code. + 4. For unknown extensions: resolve absolute path and scan first 1024 bytes for null bytes; binaries return `true`. + +**Side effect note:** `ShouldIgnoreFile` opens the file on every invocation to classify unknown extensions β€” this is a real I/O side effect per call, not cached. + +### Code Discovery Pipeline (`DiscoverCodeFiles`) + +**Input:** repository root + list of ignore patterns (strings). +**Output:** slice of relative source-code paths. + +**Algorithmic flow:** +1. Compile user-supplied ignore patterns and `.gitignore` entries into a single `GitIgnore` object via `LoadGitignore`. +2. Walk the entire tree recursively using `filepath.WalkDir`. For each entry: + - Directory: if name starts with `.` or matches any compiled pattern, skip subtree entirely. + - File: delegate to `ShouldIgnoreFile`. +3. Collect passing files into result slice. + +**Walk error handling:** If a walk callback returns an error for a single entry, the error is printed to `stderr` and that item is skipped (`return nil`). The overall `err` from `WalkDir` propagates at the end β€” terminal fatal errors are returned; non-fatal entries are swallowed silently. + +### Binary Classification (`IsBinaryFile`) + +Scans first 1024 bytes for null bytes to distinguish text source files from binary content. Returns `true` (ignored) on any I/O error other than EOF β€” this "fail-closed" assumption treats unreadable files as likely binary rather than propagating the actual error. + +**Constants:** `binaryDetectionBufSize` is lowercase and package-private; not exported. + +--- + +## Git Tools (`git_tools.go`) + +### External Command Execution (`RunGit`) + +Spawns `git --no-pager` via `exec.Command` with any additional arguments from the caller. The subprocess runs within `repoRoot` as its working directory. Stdout/stderr are captured into in-memory `bytes.Buffer`; no file, network, or DB I/O occurs. + +**Failure handling:** On error, wraps the original with stderr content: +``` +fmt.Errorf("git command failed: %v, stderr: %s", err, trimmedErr) +``` +Returns `(string, error)` β€” partial stdout is still delivered alongside the error. No panics. + +### Repository Validation (`VerifyGitRepo`) + +Delegates to `RunGit` with subcommand `rev-parse --is-inside-work-tree`. If this invocation results in an error, returns a fixed message: `"not a git repository (or any of the parent directories)"`. The original stderr is discarded at this wrapper layer β€” normalized to a single sentinel-style message for downstream callers. + +--- + +## Concurrency & State Analysis + +**Mutable state:** None detected across both files. All constants are `const` (compile-time); all variables (`safePath`, `dir`, `tmpFile`, `tmpName`, `patterns`, `files`, `buf`, `cmd`, `stdout`, `stderr`, etc.) are function-local and never escape their defining scope. No package-level global state is modified. + +**Concurrency mechanisms:** None detected. No `sync.Mutex`, channels, `atomic` types, or other synchronization primitives. Shared mutable state does not exist to protect. + +--- + +## Error Handling Summary + +| Pattern | Where | +|---|---| +| Wrap-and-return (preserve stderr) | `RunGit` | +| Normalize to sentinel message | `VerifyGitRepo` | +| Sentinel + wrap with context | `ReadFileSafely`, `WriteFileSafely`, `LoadGitignore` | +| Boolean return, resolve failures as "ignored" | `ShouldIgnoreFile` | +| Swallow per-entry walk errors, propagate terminal | `DiscoverCodeFiles` | +| Fail-closed (I/O error β†’ binary assumption) | `IsBinaryFile` | + +**No panics** observed across either file. All errors are returned to callers as Go values. \ No newline at end of file diff --git a/wiki/modules/root.md b/wiki/modules/root.md new file mode 100644 index 0000000..f1092ce --- /dev/null +++ b/wiki/modules/root.md @@ -0,0 +1,251 @@ +# Code Reducer β€” Architecture Documentation + +## System Overview + +Code Reducer is a multi-phase LLM analysis pipeline that generates technical documentation from Go source code. The system operates across four subsystems: **config**, **engine**, **security**, and **tools**. No shared mutable state exists across subsystem boundariesβ€”each module operates on caller-provided arguments and returns results without synchronization primitives. + +**Data flow**: `internal/config` resolves a fully-populated configuration via multi-source priority merging β†’ `internal/engine` consumes that config to drive an LLM pipeline β†’ `internal/security` enforces path containment for all filesystem operations the engine performs β†’ `internal/tools` provides file I/O and git discovery primitives used by both security and engine subsystems. + +## Entry Point Architecture + +The binary entry point (`main.go`) is a thin wrapper delegating all work to an external command package. Its observable surface: read an error from Execute() β†’ print it to stderr β†’ exit with code 1. The `cmd` module defines the root cobra command, registers subcommands (init, update, setup), resolves configuration from merged sources, routes engine events to stdout/stderr, handles OS signals for graceful shutdown, and delegates all substantive work to an unshown executeCommand entry point. + +**Signal handling**: A handler captures INT (os.Interrupt) and TERM (syscall.SIGTERM) via signal.NotifyContext. A deferred call ensures context cancellation when a signal arrives. The engine runner's returned error is then wrapped as `"documentation run failed: …"`. + +## Configuration Lifecycle + +### Types & Defaults + +`config.go` declares three exported types: `ExtractionStep` (per-phase identifiers carrying name string and prompt text), `Config` (aggregate configuration container referenced by all exported functions), and compile-time constants for environment variable keys, Ollama default values (`OllamaDefaultBaseURL = "http://localhost:11434"`, `OllamaDefaultModelID = "ornith:9b"`, `OllamaDefaultNumCtx = 8192`), file-system defaults (`DefaultDocsDir = "wiki"`, `ConfigFileName = ".code-reducer.yaml"`), and multiline prompt templates. + +### File Lifecycle Implementation + +`io.go` provides three exported functions: `ConfigExists(cwd string)` returns a boolean, `LoadConfig(cwd string)` returns `(*Config, error)`, and `SaveConfig(cwd string, cfg *Config)` returns an error. + +**Atomic Write Pattern for SaveConfig**: The save operation does not write directly to the target path. Sequence: marshal `cfg` into YAML bytes β†’ apply formatting normalization (collapsing single blank lines to double blank lines before known section headers: `system_prompt`, `module_synthesis_prompt`, `architecture_prompt`, `file_fact_consolidation_prompt`, `extraction_steps`, `ignore`) β†’ create a temp file matching `.tmp.*` in the same directory β†’ write normalized YAML into it β†’ sync and close β†’ set permissions via `os.Chmod` β†’ atomically rename temp over original. Readers never observe a partially-written config during save because the old file remains intact until the new one is fully ready to replace it. A deferred function calls `tmpFile.Close()` and `os.Remove(tmpName)`. On success, `Close` runs twice on the temp fileβ€”Go allows this safelyβ€”and the defer's `Remove` returns an error that is silently swallowed because the original config now holds the renamed data. + +**Load-Save Roundtrip Integrity**: Loading preserves whatever formatting was originally written; normalization is applied only on write, not read. This decouples the in-memory representation from on-disk formatting expectations. `LoadConfig` returns a fresh `*Config` via unmarshaling with no shared state. Errors from `os.ReadFile` propagate directly; YAML parse failures are wrapped with a `"failed to parse yaml config"` prefix chained via `%w`; marshal failures similarly wrap a descriptive message. + +### Multi-Source Configuration Resolution + +`resolve.go` exposes a single exported function: `ResolveConfig(repoRoot, modelIDFlag, numCtxFlag string)` returning `(*config.Config, error)`. Internal helper `mergeAndDeduplicate` is lowercase and not part of the public surface. Only usage (not definition) of package-level types (`Config`, `OllamaDefaultModelID`, `CodeReducerModelIDEnvKey`, `DefaultExtractionSteps`) is visible here. + +**Resolution Priority System**: Each configurable field follows its own independent priority chain, lowest to highest: **defaults β†’ YAML config file β†’ environment variables β†’ CLI flags**. The merge-and-deduplicate logic operates on local function scopes with no cross-goroutine visibility and no shared mutable state. + +**Field-Specific Priority Chains**: `ModelID` applies Default > YAML > Env Var > CLI Flag. `OllamaBaseURL` applies Default > YAML > Env Var. `OllamaNumCtx` (context size) applies Default > YAML > Env Var > CLI Flag. `DocsDir` applies Default > YAML. System prompts (`SystemPrompt`, `ModuleSynthesisPrompt`, `ArchitecturePrompt`, `FileFactConsolidationPrompt`) all start with default, then override with YAML if non-empty. + +**Validation Rules**: Numeric fields: CLI flag and env var values are parsed as integers; a value must be greater than zero to take effect. Invalid or non-positive values fall through silently. String fields: empty string is treated as "not set" β€” the lower-priority default remains active. Config file absence (`config.yml`): not an error; if missing, an empty `Config{}` struct is created and defaults are applied. Other parse errors are surfaced. + +**Algorithmic Flow for ResolveConfig**: (1) Load YAML config β€” if it exists and parses successfully; otherwise start with an empty struct (missing-file case). (2) Resolve extraction steps β€” use YAML value if present, else fall back to `DefaultExtractionSteps`. (3) Deduplicate ignore list from the loaded config using a map-based seen-set while preserving first-seen order. (4) For each field, apply its specific priority chain: start with built-in default constant β†’ override with YAML (if non-empty) β†’ override with environment variable (if set and valid for numeric types) β†’ override with CLI flag (highest priority; numeric types require successful integer parse and positive value). (5) Return the fully resolved `Config` struct, or an error if the YAML file fails to parse (excluding missing-file case). + +**Error Handling in ResolveConfig**: When `os.IsNotExist(err)` is true during config loading, defaults apply with no error returned. Any other error (parse failure, permission denied) is wrapped with a prefix and returned as `(*Config, nil, error)` β€” the config pointer is `nil`. Silent branches: string comparisons (`!= ""`, `> 0`) short-circuit without errors. `strconv.Atoi` failures are caught inline; env/flag values simply not used. Map lookups for deduplication have no error path. + +**Always-On Return**: The function always returns a non-nil tuple `(*Config, error)` β€” either a populated config with `nil` error, or a `nil` config with an error string. No panics anywhere in this file. + +## Engine Pipeline Architecture + +### Constants & Types + +`constants.go` declares compile-time constants: `defaultHTTPTimeout` (10 min), `maxErrorBodyBytes` (1024 bytes), `defaultChunkOverlap` (800 chars), `minNumCtxFloor` (512 tokens), `contextWindowAllocRatio` (0.75), `maxCharsMultiplier` (3), `metadataFileName` (`.metadata.json`), `agentsFileName` (`AGENTS.md`), `defaultDirPerm` (0755). Type alias `LogEventFunc` carries signature `func(EventType, string)` with no business rules defined here. + +### LLM Client Communication + +Unexported except `Message` struct (`Role string`, `Content string`). The package-private `llmClient` carries: model ID, base URL, context count, HTTP client pointer. All set once during construction; never modified afterward within this file. No mutexes or channels used. + +**Algorithmic Flow**: (1) Prepare payload: assemble request body by prepending `systemPrompt` as a system-role message before any user-supplied messages, serialize as JSON to Ollama `/api/chat`. (2) Send HTTP request: single synchronous POST with fixed timeout; no retries (fail-fast policy). (3) Handle response: on 200 OK β†’ deserialize body β†’ extract `message.content` field. Non-2xx status code β†’ error string containing both HTTP status and raw response payload. (4) Return result: extracted text content or wrapped error describing what went wrong. + +**Error Handling Patterns**: Marshal failure wraps via `%w` for caller inspection via `errors.Is`. HTTP request construction returns unwrapped; context lost for callers. Network/transport error returns unwrapped; timeout vs DNS vs connect indistinguishable to caller. Success-path body read returns unwrapped (rare). JSON unmarshal failure wraps via `%w`. Non-OK status response uses custom message discarding read error (`_`); uses undefined `maxErrorBodyBytes` constant. No retry logic. + +### Runner & Entry Point + +`Runner` wraps a config pointer and exposes a single `Run(ctx, repoRoot, mode, onEvent)` method that is the only public entry point into the pipeline. It performs four sequential operations: (1) Gitignore lockfile maintenance β€” attempts to append the lockfile path to `.gitignore`. Failure is logged via `onEvent(EventStatus, ...)` and execution continues; no error returned. (2) Repository lock acquisition β€” calls `security.AcquireLock(repoRoot)`. On failure, returns wrapped error: `"failed to acquire repository lock: %w"`. No partial work proceeds without the lock. (3) Mode dispatch β€” constructs an LLM client from config fields (`cfg.ModelID`, `cfg.OllamaBaseURL`, `cfg.OllamaNumCtx`) and instantiates an `orchestrator`. Dispatches to either `RunInit` or `RunUpdate`; any other mode returns `"unsupported mode: %s"`. (4) Deferred lock release β€” `defer lock.Unlock()` runs after the pipeline completes, regardless of success or failure. + +The `Runner.cfg` field is set once in `NewRunner(cfg)` and never mutated within this file. The pointer type allows external reassignment but no such mutation occurs here. + +### Orchestrator & Pipeline Modes + +Unexported struct with two exported constants on the `EventType` type alias: `EventStatus = "status"` and `EventError = "error"`. The unexported `orchestrator` struct carries a method receiver; its public surface is empty. + +**RunInit β€” Cold Start Flow**: (1) setupPipeline() β†’ discover files, compute hashes, load .gitignore (swallowed), load cache (swallowed). (2) Mark every directory as affected (full regeneration). (3) Synthesize root summary from full tree via LLM calls. (4) Generate architecture.md + quickstart.md using root summary. (5) Write agent guidelines file (append if marker missing, overwrite otherwise). (6) Persist cache to disk β†’ complete. + +**RunUpdate β€” Incremental Refresh Flow**: (1) setupPipeline() β†’ discover current files, compute hashes. (2) If extraction steps changed β†’ invalidate entire cache (.StepsHash mismatch resets Files and Modules maps). (3) Classify all discovered files into Added/Modified/Deleted buckets by comparing current vs cached SHA-256 hashes. (4) Build directory tree from allowed (existing + new) files only. (5) Purge stale module summaries: for any `cache.Modules` path not represented in the live tree, delete both the cache entry and the physical `.md` file (`_ = os.Remove(...)` β€” error swallowed). (6) Compute affected directories; propagate upward through parent dirs. (7) If `len(affectedDirs) == 0` β†’ return early (no-op, docs are up to date). (8) Synthesize root summary from the affected region via LLM calls. (9) Re-generate architecture.md + quickstart.md only if `.` is in `affectedDirs`, or either file does not exist on disk, or resolution fails (treated as "not exists"). (10) Persist cache β†’ complete. + +**Pipeline Context**: Unexported struct carrying: LLM client pointer, repo root string, config pointer, `*MetadataCache` pointer, affected directory set map, precomputed file hashes map, and event logger callback. Constructed fresh per invocation; no shared instances exist across goroutines in this file. + +### Cache Persistence Layer + +Two structs with no methods: **`FileCacheEntry`** β€” pairs a SHA256 digest string (`sha256`) with extracted facts string (`facts`). Stored in `MetadataCache.Files map[string]FileCacheEntry`. **`MetadataCache`** β€” carries version int, steps hash string, files map, and modules map (module dir β†’ synthesized summary). + +**IsInitialized(repoRoot, docsDir) bool**: Returns true only if the cache file exists AND is readable in one call. Reuses the existence check from `loadMetadataCache`. No error return. + +**saveMetadataCache(repoRoot, docsDir, cache) error**: Serializes to `{docsDir}/{metadataFileName}` with indented JSON (`json.MarshalIndent(cache, "", " ")`). Marshaling errors propagate directly; write errors pass through `tools.WriteFileSafely`. The version field is pinned to the current value on every save. + +**Version Compatibility Guard**: On load: if `cache.Version != currentCacheVersion` (currently 1), returns a fresh zero-valued `MetadataCache{}` with nil error. No warning, no panicβ€”caller sees success with an empty cache. This prevents stale metadata from corrupting future runs without explicit migration handling. + +**Fault-Tolerant Initialization**: If the file does not exist (`os.ErrNotExist` sentinel matched via `errors.Is`), returns a populated zero-valued cache with nil error. If it exists but cannot be read or unmarshaled, returns wrapped error `"failed to read metadata cache: %w"`. Callers can distinguish "not yet initialized" from "corrupt data." + +### Chunking & Reduction Engine + +All unexported; package-private. Implements a **context-aware recursive reduction** for LLM-based synthesis: (1) Split: Input list divided into batches whose combined character count fits within `contextSize Γ— maxCharsMultiplier`. (2) Reduce each batch: Each batch sent to LLM via domain-specific prompt β†’ one summary string per batch. If only one batch exists, returned directly; otherwise summaries concatenated and fed back into step 1 recursively until a single result remains. (3) Graceful truncation: Items exceeding `maxChars` are truncated with `"...\n...[truncated]"`. If every item exceeds the per-item budget (`allowedPerItem`), all items uniformly truncated to avoid infinite loops. (4) Context cancellation: Checked at entry points and before each recursive reduction round; cancellation errors propagate as-is. + +**chunkTextWithOverlap**: Text-splitting utility with overlap between adjacent chunks. Appears available for reuse elsewhere but not referenced by any function here. + +### Synthesis Core + +All types and functions unexported (`pipelineContext`, `synthesizeNode`). Package-private. + +**Recursive Directory Synthesis**: A directory's summary is derived by recursively synthesizing children (subdirectories) and files together. Final result stored in cache under `cache.Modules[node.Path]` and written to disk at `docs/modules/`. + +**Caching Strategy with Hash-Based Invalidation**: Module-level: Stores synthesized summary per directory path. Reused when affected status indicates no changes (short-circuit return). File-level: Stores SHA256 hash + extracted facts in `cache.Files[f]`. Cache hit determined by comparing precomputed or read file hash against cached entry; only re-reads on miss. + +**Multi-Step Extraction Pipeline Per File Chunk**: (1) System prompt augmented with current extraction step's prompt. (2) Single LLM call extracts facts from chunk β†’ one fact per step. (3) All steps produce respective facts for the same file β†’ consolidated via `reduceFileFacts`. + +**Synthesis Order & Assembly**: (1) File facts (sorted child name order). (2) Child subdirectory summaries (after all files processed). (3) Combined list reduced to final summary via `reduceInChunks` at directory level. + +**Affected-Dir Pruning**: If a directory path has no affected status AND a cached module summary exists β†’ short-circuit return cached value without reprocessing children or files. + +### Tree & Change Detection + +Exported types: **`FileChange`** β€” `Path string`, `Status string` ("Added", "Modified", "Deleted"). **`DirNode`** β€” `Path string`, `Files []string`, `Children map[string]*DirNode`. + +**Three-Phase Algorithm**: (1) Build tree: Given raw file paths, split each on `/`, create intermediate `DirNode` entries for subdirectories, attach files to leaf nodes via `buildTree`. (2) Determine affected directories from changes: For each input change, mark directly changed file as affected (if exists in current set). When a file is Deleted, mark parent directory as affected. Walk recursively and check additional conditions: if node path matches `docs/modules/` but no such doc exists on disk β†’ affected; if node has empty cached metadata entry β†’ affected. (3) Propagate status upward: Recursive walk of `DirNode` tree propagates any `affected` flag from children to parents so a parent is considered affected if any child is. + +### JSON Parser Utility + +**stripOuterMarkdownFence(input) string**: Single-pass parser that strips markdown or JSON code fences from strings: (1) Trims whitespace from input. (2) Attempts regex match against pattern capturing triple backticks with 3+ chars, optionally followed by `markdown` or `json`, then all content between opening and closing fences (group 1). (3) If matched β†’ returns trimmed captured inner content. (4) If no match β†’ returns original trimmed input unchanged. No error return parameter. `regexp.MustCompile` panics on invalid pattern at package initβ€”unhandled within this file, propagates to any caller importing the package. + +### Utility Functions for Filename Generation and Event Logging + +This file provides two unexported utility functions supporting the engine's documentation generation pipeline and optional event logging instrumentation. Neither function communicates with external systems, modifies shared state, or returns errors. All operations are pure in-memory transformations. + +**toSafeMarkdownFilename β€” Module Path to Markdown Filename Conversion**: Transforms a module path into a safe markdown documentation filename, enforcing the engine's implicit one-to-one mapping contract between modules and their `.md` documentation files. The root module is special-cased to default to `root.md`. Data flow: (1) Input: a raw string representing a module path (e.g., `"internal/engine"`). (2) Character substitution: every `/` character in the input is replaced with `_`. (3) Extension appended: `.md` is concatenated to produce the final filename. (4) Output: a single `string` containing the resulting markdown filename. No filesystem, network, database, or other external I/O occurs during execution. The function returns only a result string; no error type is returned. Invalid input characters (anything other than `/`) are passed through unmodified β€” callers receive no escape hatch for detecting malformed paths. Pure: no mutable state, no package-level globals modified, no struct fields touched. + +**makeLogEvent β€” Optional Event Callback Adapter**: Wires an optional callback into a logging pipeline. When invoked with an event type and message pair, it either forwards the call to the registered handler or silently discards it. This makes the logging subsystem purely additive β€” no required consumer exists for the engine's log events; instrumentation is opt-in only. Data flow: (1) Input: a user-provided `onEvent` callback of type `LogEventFunc`. (2) Closure construction: returns a new function accepting `(EventType, message)` pairs. (3) Invocation behavior: if `onEvent == nil`, the returned closure does nothing when called β€” no panic, no error, no log output. If `onEvent` is non-nil, the call is forwarded to it with the provided type and message arguments. No I/O anywhere in this flow. Pure: no mutable state, no package-level globals modified. Silent failure path: when `onEvent == nil`, events are swallowed without notice or error wrapping. The returned function is always valid (non-nil), so callers can invoke it safely regardless of whether a handler was registered. + +**Referenced Types**: The following types are referenced in this file but not defined within it: `EventType` β€” categorizes log events into discrete types; carried as the first argument to event handlers and consumed by the returned closure from `makeLogEvent`. `LogEventFunc` β€” function signature for user-provided event handlers. Expected to accept `(EventType, message string)` or equivalent parameters. `Event` (struct) β€” domain model with at least two fields: `Type` (`EventType`) and `Message` (`string`). Events are tracked as discrete, typed records carrying descriptive payloads. + +**Concurrency & State Summary**: Mutable state: None. Both functions operate entirely on their inputs and return values without touching package-level variables or struct fields. Concurrency mechanisms: None. No mutexes, channels, atomic operations, or synchronization primitives are used in this file. + +## Security Invariants + +### Module Responsibility + +This module enforces two non-negotiable invariants for any code-reducer session: (1) the working directory must remain strictly within the repository root boundary, and (2) only one process may hold an exclusive lock on the shared resource at any given moment. It also maintains a `.gitignore` entry that excludes the lockfile from version control while ensuring it is automatically tracked if absent. + +The module communicates with the outside world indirectly: `security.go` performs filesystem I/O and returns sentinel error values to callers; those sentinels are defined in `errors.go`. No panic recovery, no error wrapping beyond `%w`, no silent swallowing of non-trivial failures occurs within these files. + +### Sentinel Error Contract + +Package-level variables: **`ErrPathTraversal`** (`error`) β€” returned when a resolved path escapes outside the repository root boundary. **`ErrLockHeld`** (`error`) β€” returned when another code-reducer process holds an exclusive file lock on the shared resource. Both variables are initialized once via `errors.New` and remain read-only throughout their scope. No mutable state, no concurrency primitives, no panic handling exists in this file. Callers detect violations by matching against these sentinels using `errors.Is(err, ErrPathTraversal)` or equivalent patterns; because the errors are unwrapped as-is (no wrapping), identity is preserved directly through the error chain. + +**Domain Concepts**: Path Traversal Violation (`ErrPathTraversal`) β€” signals when a resolved path escapes outside the repository root boundary. This is a classic path traversal / directory escape guardrail. Lock Conflict (`ErrLockHeld`) β€” signals when another process already holds an exclusive file lock, indicating contention on a shared resource. + +### Lock Acquisition and Release + +**Struct Definition**: `SimpleLock` carries: `lockPath string` (absolute path to the lockfile on disk), `file *os.File` (open file descriptor for the lockfile, or nil if not acquired), `mu sync.Mutex` (protects internal struct fields during `Unlock()`), `closed bool` (marks whether the lock has been released and the file descriptor closed). + +**`*SimpleLock.Unlock()`**: Closes the lock file descriptor and removes the lockfile in a thread-safe manner. **Idempotency contract**: Calling `Unlock()` on an already-closed lock returns `nil`. This is intentional, not accidental. Error handling inside the method: If `l.closed == true` β†’ return `nil`. Close the file descriptor (`os.File.Close`) β€” error captured in local variable; if close fails, set `file = nil` so a subsequent close becomes a no-op. Remove the lockfile via `os.Remove`. If both close and remove fail: remove error overwrites prior close error only when close returned without error. If close failed first, any remove error is ignored. This means the final returned error reflects whichever operation last produced an error, not necessarily the most severe one β€” an intentional design choice worth flagging for callers. If `os.IsNotExist(err)` on remove β†’ silently accepted as normal idempotent cleanup. + +### Path Resolution + +**Signature**: `func SafeResolve(repoRoot, inputPath string) (string, error)`. **Responsibility**: Resolve any input path strictly inside the repository root. The function performs a multi-step containment check to defeat symlink-based escape vectors and physical directory traversal. + +**Algorithmic Steps**: (1) Call `filepath.Abs` on `inputPath`. (2) Evaluate symlinks on the existing ancestor via `EvalSymlinks` β€” prevents symlink-based escape from parent directories that exist but are symbolic links. (3) Walk up from the target until a physically-existing directory is found, then re-evaluate symlinks on that ancestor via `EvalSymlinks` again. (4) Rejoin with the remaining suffix components after the physical ancestor boundary. (5) Verify final result is under `resolvedRoot` via relative path check (`filepath.Rel`). + +**Error Handling**: Any error from `Abs`, first `EvalSymlinks`, second `EvalSymlinks`, or `Lstat` (non-exist) wraps with descriptive message using `fmt.Errorf(..., err)` β€” underlying cause recovered via `%w`. Unrecognized error from `Lstat` (`!os.IsNotExist`) returns immediately as wrapped error. Path escapes repo root (`..` prefix or non-nil rel) returns sentinel-style error wrapping `ErrPathTraversal` with the original input path. + +### Lock Acquisition + +**Signature**: `func AcquireLock(repoRoot string) (*SimpleLock, error)`. **Responsibility**: Establish an exclusive process-level lock on a shared resource. Uses `O_EXCL` flag for atomic creation of the lockfile β€” if the lockfile already exists, acquisition fails immediately without silent success. + +**Lockfile Lifecycle**: Resolve path calls `SafeResolve(repoRoot)` to obtain absolute lock path. Create atomically opens file with `O_WRONLY | O_CREATE | O_EXCL` β€” atomic exclusive creation prevents race between existence check and open. Record identity writes PID of the acquiring process to the newly created lockfile. Failure cleanup on failure: closes file, removes lockfile. + +**Error Handling**: `SafeResolve` fails propagates as-is to caller. File open fails AND error is `os.IsExist` (lock already held) returns wrapped sentinel error wrapping `ErrLockHeld`, including the lock path β€” caller can detect stale-lock condition specifically via `errors.Is(err, ErrLockHeld)`. File open fails with any other error wraps: `"failed to acquire lock at %s: "`. Write PID fails file closed, lockfile removed (cleanup), wrapped error returned. + +### Gitignore Maintenance + +**Signature**: `func EnsureGitignoreHasLockfile(repoRoot string) (error)`. **Responsibility**: Maintain the convention that the lockfile is excluded from version control but automatically tracked if missing. Appends the lockfile entry to `.gitignore` only if not already present; uses atomic rename via temp file + sync before close to prevent partial writes. + +**Algorithmic Steps**: (1) Resolve `.gitignore` path via `SafeResolve(repoRoot)`. (2) Read existing `.gitignore` content (`os.ReadFile`). (3) Parse lines, check if lockfile entry already exists. (4) If not present: construct new content with the lockfile entry appended (with trailing-newline normalization). (5) Create a temp file in same directory as `.gitignore`. (6) Write to temp file β†’ call `Sync()` on it β†’ close it β†’ `Chmod` it. (7) Rename temp file over original `.gitignore`. + +**Error Handling**: `SafeResolve` fails propagates as-is. Read fails AND not `os.IsNotExist` (file exists but read error) wraps: `"error reading .gitignore: "` β€” caller must handle missing-file vs I/O-error distinction separately. CreateTemp fails wraps with descriptive message including `%w`. Write to temp file fails wraps; temp file is NOT cleaned up (defer handles close + remove, but write error returns before cleanup path executes). Sync fails wraps β€” note that `os.Chmod` and subsequent rename may have already run on a partially-written temp file if this error occurs late in the sequence. Close fails wraps. Chmod fails wraps. Rename (atomic swap) fails wraps with descriptive message. + +The deferred function (`tmpFile.Close()` + `os.Remove(tmpName)`) runs regardless of return path. Because the function returns before the defer has a chance to execute in some failure paths, cleanup may not happen for intermediate states β€” but since we're using temp file with unique name and same-directory rename, the OS-level temp file is cleaned by `os.Remove` in the defer on any error return path. If *write* fails mid-stream before close/rename, the deferred Remove will clean up the partially-written temp file. + +### Data Flow Summary + +``` +[Initialize] β†’ [Ensure path safety for all repo-root operations (SafeResolve)] β†’ +[If no process holds the lock] β†’ [Create lockfile with PID via O_EXCL (AcquireLock)] β†’ +[Run business logic within locked session] β†’ +[Append entry to .gitignore if needed (EnsureGitignoreHasLockfile)] +``` + +The two non-obvious constraints are: (1) path resolution walks symlinked ancestors before rejoining suffix components, and (2) gitignore updates use atomic rename with explicit sync before close. + +**Key Patterns Observed Across the Module**: Error wrapping throughout β€” All functions use `fmt.Errorf(..., err)` with `%w`, making root-cause recovery possible via `errors.Is()`. Custom sentinel errors β€” At least two custom error variables referenced: `ErrPathTraversal` and `ErrLockHeld`. These enable typed error detection by callers without requiring unwrapping logic on this file's side. No panic anywhere β€” All failure paths return to caller with an error value. Idempotent operations deliberately silent β€” `Unlock()` on already-closed lock returns nil; missing `.gitignore` in EnsureGitignoreHasLockfile returns nil. These are design choices, not accidentals. Atomic rename pattern for .gitignore β€” Write to temp β†’ Sync β†’ Close β†’ Chmod β†’ Rename is the standard atomic-update pattern. The `Sync()` call before close is explicit and unusual (normally deferred), but here it's called eagerly before close in this implementation. No global package-level mutable state exists in either file β€” all state is either local to a function or encapsulated within the `SimpleLock` receiver, which is accessed only through its methods. + +## Tools Subsystem + +### Module Responsibility + +The `internal/tools` package provides repository-scanning primitives for a codebase-analysis tool. It exposes two functional domains: **safe local file I/O** (`file_tools.go`) and **git subprocess orchestration** (`git_tools.go`). Together they enable the caller to discover source-code files, classify their content type, read/write artifacts safely, and validate that the target directory is a git work-tree before proceeding. + +Data flow follows a two-phase pattern: (1) `VerifyGitRepo` (or an equivalent pre-flight check) confirms the working directory corresponds to a valid git repository by invoking `git rev-parse --is-inside-work-tree`. (2) Once validated, `DiscoverCodeFiles` walks the entire tree under that root, applying layered ignore rules and returning only text-source files. Both domains share these architectural constraints: **no panics**, **error wrapping** (stderr is preserved for diagnostics), and **no network/DB calls**. All I/O operates within a single local process or on the local filesystem. + +### File Operations + +**Safe Read / Write Primitive**: `ReadFileSafely` and `WriteFileSafely` implement TOCTOU-safe file operations using the temp-file + rename pattern. Shared sequence: (1) Resolve a virtual path to an absolute filesystem path via `SafeResolve`. (2) On write: create parent directories with `defaultDirPerm`; on read: no directory creation. (3) Open or create a temporary file in the target directory using `os.CreateTemp`. (4) Write content (for writes) or read from the resolved path (for reads). (5) Close and sync (writes only); close (reads only). (6) Rename temp into final destination for writes; return contents for reads. + +**Symlink detection**: Both paths verify the target is not a symlink via `os.Lstat`/`f.Stat`. Pre-open checks prevent TOCTOU swap attacks where an attacker replaces a real file with a symlink between resolution and open. Post-open checks catch any state changes after the initial open. + +**Error semantics**: `ReadFileSafely`: returns `nil, err` on failure; wraps each step with context strings (`failed to open file`, `failed to lstat`, etc.). Symlink detection yields an explicit "security violation" error rather than a generic I/O error. `WriteFileSafely`: all failure paths return wrapped errors (directory creation, temp write, sync, close, chmod, rename). No panic usage; no sentinel values. Deferred cleanup: Both functions defer `f.Close()` for reads and use deferred `os.Remove` in `WriteFileSafely` to clean up temp files if the rename fails. + +**Gitignore Handling**: **`LoadGitignore(repoRoot)`**: Opens `.gitignore` under `repoRoot`. If absent (`os.IsNotExist`), returns `nil, nil` β€” no error raised. Any other open/read failure is returned unwrapped. **`ShouldIgnoreFile(repoRoot, relPath, gitIgnore)`**: Applies layered rules: (1) Check against compiled `GitIgnore` patterns (user-supplied + `.gitignore`). (2) Reject any path component starting with `.` or ending with `.egg-info`, regardless of other rules. (3) If extension matches a known text-source language list, accept as code. (4) For unknown extensions: resolve absolute path and scan first 1024 bytes for null bytes; binaries return `true`. + +**Side effect note**: `ShouldIgnoreFile` opens the file on every invocation to classify unknown extensions β€” this is a real I/O side effect per call, not cached. + +### Code Discovery Pipeline + +**Input**: repository root + list of ignore patterns (strings). **Output**: slice of relative source-code paths. + +**Algorithmic flow**: (1) Compile user-supplied ignore patterns and `.gitignore` entries into a single `GitIgnore` object via `LoadGitignore`. (2) Walk the entire tree recursively using `filepath.WalkDir`. For each entry: Directory: if name starts with `.` or matches any compiled pattern, skip subtree entirely. File: delegate to `ShouldIgnoreFile`. (3) Collect passing files into result slice. + +**Walk error handling**: If a walk callback returns an error for a single entry, the error is printed to `stderr` and that item is skipped (`return nil`). The overall `err` from `WalkDir` propagates at the end β€” terminal fatal errors are returned; non-fatal entries are swallowed silently. + +### Binary Classification + +**IsBinaryFile**: Scans first 1024 bytes for null bytes to distinguish text source files from binary content. Returns `true` (ignored) on any I/O error other than EOF β€” this "fail-closed" assumption treats unreadable files as likely binary rather than propagating the actual error. + +**Constants**: `binaryDetectionBufSize` is lowercase and package-private; not exported. + +### Git Tools + +**External Command Execution (`RunGit`)**: Spawns `git --no-pager` via `exec.Command` with any additional arguments from the caller. The subprocess runs within `repoRoot` as its working directory. Stdout/stderr are captured into in-memory `bytes.Buffer`; no file, network, or DB I/O occurs. + +**Failure handling**: On error, wraps the original with stderr content: +``` +fmt.Errorf("git command failed: %v, stderr: %s", err, trimmedErr) +``` +Returns `(string, error)` β€” partial stdout is still delivered alongside the error. No panics. + +**Repository Validation (`VerifyGitRepo`)**: Delegates to `RunGit` with subcommand `rev-parse --is-inside-work-tree`. If this invocation results in an error, returns a fixed message: `"not a git repository (or any of the parent directories)"`. The original stderr is discarded at this wrapper layer β€” normalized to a single sentinel-style message for downstream callers. + +### Concurrency & State Analysis + +**Mutable state**: None detected across both files. All constants are `const` (compile-time); all variables (`safePath`, `dir`, `tmpFile`, `tmpName`, `patterns`, `files`, `buf`, `cmd`, `stdout`, `stderr`, etc.) are function-local and never escape their defining scope. No package-level global state is modified. + +**Concurrency mechanisms**: None detected. No `sync.Mutex`, channels, `atomic` types, or other synchronization primitives. Shared mutable state does not exist to protect. + +### Error Handling Summary + +| Pattern | Where | +|---|---| +| Wrap-and-return (preserve stderr) | `RunGit` | +| Normalize to sentinel message | `VerifyGitRepo` | +| Sentinel + wrap with context | `ReadFileSafely`, `WriteFileSafely`, `LoadGitignore` | +| Boolean return, resolve failures as "ignored" | `ShouldIgnoreFile` | +| Swallow per-entry walk errors, propagate terminal | `DiscoverCodeFiles` | +| Fail-closed (I/O error β†’ binary assumption) | `IsBinaryFile` | + +**No panics** observed across either file. All errors are returned to callers as Go values. \ No newline at end of file diff --git a/wiki/quickstart.md b/wiki/quickstart.md new file mode 100644 index 0000000..a03fc0a --- /dev/null +++ b/wiki/quickstart.md @@ -0,0 +1,189 @@ +# Code Reducer β€” Quick Start Guide + +## System Overview + +Code Reducer is a multi-phase LLM analysis pipeline that generates technical documentation from Go source code. The system spans four subsystems: **config**, **engine**, **security**, and **tools**. No shared mutable state exists across subsystem boundariesβ€”each module operates on caller-provided arguments and returns results without synchronization primitives. + +## Entry Point + +The binary entry point (`main.go`) is a thin wrapper that reads an error from `Execute()`, prints it to stderr, and exits with code 1. The `cmd` module defines the root cobra command, registers subcommands (init, update, setup), resolves configuration from merged sources, routes engine events to stdout/stderr, handles OS signals for graceful shutdown (INT + TERM via `signal.NotifyContext`), and delegates all substantive work to an unshown executeCommand entry point. + +## Configuration Lifecycle + +### Resolution Priority Chain + +Each configurable field follows its own independent priority chain, lowest to highest: **defaults β†’ YAML config file β†’ environment variables β†’ CLI flags**. The function always returns a non-nil tuple `(*Config, error)`. Config file absence is not an error; if missing, an empty struct is created and defaults are applied. Parse failures propagate as errors. + +### Atomic Write Pattern for SaveConfig + +The save operation does not write directly to the target path. Sequence: marshal into YAML bytes β†’ apply formatting normalization (collapsing single blank lines to double blank lines before known section headers) β†’ create a temp file matching `.tmp.*` in the same directory β†’ write normalized YAML into it β†’ sync and close β†’ set permissions via `os.Chmod` β†’ atomically rename temp over original. A deferred function calls `tmpFile.Close()` and `os.Remove(tmpName)`. + +### Multi-Source Configuration Resolution + +The public surface is a single exported function: `ResolveConfig(repoRoot, modelIDFlag, numCtxFlag string)` returning `(*config.Config, error)`. Internal helper `mergeAndDeduplicate` is lowercase. Only usage (not definition) of package-level types (`Config`, `OllamaDefaultModelID`, `CodeReducerModelIDEnvKey`, `DefaultExtractionSteps`) is visible here. + +**Field-Specific Priority Chains**: +- `ModelID`: Default > YAML > Env Var > CLI Flag +- `OllamaBaseURL`: Default > YAML > Env Var +- `OllamaNumCtx` (context size): Default > YAML > Env Var > CLI Flag +- `DocsDir`: Default > YAML +- System prompts (`SystemPrompt`, `ModuleSynthesisPrompt`, `ArchitecturePrompt`, `FileFactConsolidationPrompt`): all start with default, then override with YAML if non-empty + +**Validation Rules**: Numeric fields: CLI flag and env var values are parsed as integers; a value must be greater than zero to take effect. Invalid or non-positive values fall through silently. String fields: empty string is treated as "not set" β€” the lower-priority default remains active. + +## Engine Pipeline Architecture + +### Runner & Entry Point + +`Runner` wraps a config pointer and exposes a single `Run(ctx, repoRoot, mode, onEvent)` method that is the only public entry point into the pipeline. It performs four sequential operations: (1) Gitignore lockfile maintenance β€” attempts to append the lockfile path to `.gitignore`. Failure is logged via `onEvent(EventStatus, ...)` and execution continues; no error returned. (2) Repository lock acquisition β€” calls `security.AcquireLock(repoRoot)`. On failure, returns wrapped error: `"failed to acquire repository lock: %w"`. No partial work proceeds without the lock. (3) Mode dispatch β€” constructs an LLM client from config fields (`cfg.ModelID`, `cfg.OllamaBaseURL`, `cfg.OllamaNumCtx`) and instantiates an orchestrator. Dispatches to either `RunInit` or `RunUpdate`; any other mode returns `"unsupported mode: %s"`. (4) Deferred lock release β€” `defer lock.Unlock()` runs after the pipeline completes, regardless of success or failure. + +### Orchestrator & Pipeline Modes + +**RunInit β€” Cold Start Flow**: +1. setupPipeline() β†’ discover files, compute hashes, load .gitignore (swallowed), load cache (swallowed) +2. Mark every directory as affected (full regeneration) +3. Synthesize root summary from full tree via LLM calls +4. Generate architecture.md + quickstart.md using root summary +5. Write agent guidelines file (append if marker missing, overwrite otherwise) +6. Persist cache to disk β†’ complete + +**RunUpdate β€” Incremental Refresh Flow**: +1. setupPipeline() β†’ discover current files, compute hashes +2. If extraction steps changed β†’ invalidate entire cache (.StepsHash mismatch resets Files and Modules maps) +3. Classify all discovered files into Added/Modified/Deleted buckets by comparing current vs cached SHA-256 hashes +4. Build directory tree from allowed (existing + new) files only +5. Purge stale module summaries: for any `cache.Modules` path not represented in the live tree, delete both the cache entry and the physical `.md` file (`_ = os.Remove(...)` β€” error swallowed) +6. Compute affected directories; propagate upward through parent dirs +7. If `len(affectedDirs) == 0` β†’ return early (no-op, docs are up to date) +8. Synthesize root summary from the affected region via LLM calls +9. Re-generate architecture.md + quickstart.md only if `.` is in `affectedDirs`, or either file does not exist on disk, or resolution fails (treated as "not exists") +10. Persist cache β†’ complete + +### Cache Persistence Layer + +Two structs with no methods: **`FileCacheEntry`** β€” pairs a SHA256 digest string (`sha256`) with extracted facts string (`facts`). Stored in `MetadataCache.Files map[string]FileCacheEntry`. **`MetadataCache`** β€” carries version int, steps hash string, files map, and modules map (module dir β†’ synthesized summary). + +**Version Compatibility Guard**: On load: if `cache.Version != currentCacheVersion` (currently 1), returns a fresh zero-valued `MetadataCache{}` with nil error. No warning, no panicβ€”caller sees success with an empty cache. This prevents stale metadata from corrupting future runs without explicit migration handling. + +**Fault-Tolerant Initialization**: If the file does not exist (`os.ErrNotExist` sentinel matched via `errors.Is`), returns a populated zero-valued cache with nil error. If it exists but cannot be read or unmarshaled, returns wrapped error `"failed to read metadata cache: %w"`. Callers can distinguish "not yet initialized" from "corrupt data." + +### Chunking & Reduction Engine + +All unexported; package-private. Implements a **context-aware recursive reduction** for LLM-based synthesis: (1) Split: Input list divided into batches whose combined character count fits within `contextSize Γ— maxCharsMultiplier`. (2) Reduce each batch: Each batch sent to LLM via domain-specific prompt β†’ one summary string per batch. If only one batch exists, returned directly; otherwise summaries concatenated and fed back into step 1 recursively until a single result remains. (3) Graceful truncation: Items exceeding `maxChars` are truncated with `"...\n...[truncated]"`. If every item exceeds the per-item budget (`allowedPerItem`), all items uniformly truncated to avoid infinite loops. (4) Context cancellation: Checked at entry points and before each recursive reduction round; cancellation errors propagate as-is. + +### Synthesis Core + +All types and functions unexported (`pipelineContext`, `synthesizeNode`). Package-private. + +**Recursive Directory Synthesis**: A directory's summary is derived by recursively synthesizing children (subdirectories) and files together. Final result stored in cache under `cache.Modules[node.Path]` and written to disk at `docs/modules/`. + +**Caching Strategy with Hash-Based Invalidation**: Module-level: Stores synthesized summary per directory path. Reused when affected status indicates no changes (short-circuit return). File-level: Stores SHA256 hash + extracted facts in `cache.Files[f]`. Cache hit determined by comparing precomputed or read file hash against cached entry; only re-reads on miss. + +**Multi-Step Extraction Pipeline Per File Chunk**: (1) System prompt augmented with current extraction step's prompt. (2) Single LLM call extracts facts from chunk β†’ one fact per step. (3) All steps produce respective facts for the same file β†’ consolidated via `reduceFileFacts`. + +**Synthesis Order & Assembly**: (1) File facts (sorted child name order). (2) Child subdirectory summaries (after all files processed). (3) Combined list reduced to final summary via `reduceInChunks` at directory level. + +### Tree & Change Detection + +Exported types: **`FileChange`** β€” `Path string`, `Status string` ("Added", "Modified", "Deleted"). **`DirNode`** β€” `Path string`, `Files []string`, `Children map[string]*DirNode`. + +**Three-Phase Algorithm**: (1) Build tree: Given raw file paths, split each on `/`, create intermediate `DirNode` entries for subdirectories, attach files to leaf nodes via `buildTree`. (2) Determine affected directories from changes: For each input change, mark directly changed file as affected (if exists in current set). When a file is Deleted, mark parent directory as affected. Walk recursively and check additional conditions: if node path matches `docs/modules/` but no such doc exists on disk β†’ affected; if node has empty cached metadata entry β†’ affected. (3) Propagate status upward: Recursive walk of `DirNode` tree propagates any `affected` flag from children to parents so a parent is considered affected if any child is. + +## Security Invariants + +### Module Responsibility + +This module enforces two non-negotiable invariants for any code-reducer session: (1) the working directory must remain strictly within the repository root boundary, and (2) only one process may hold an exclusive lock on the shared resource at any given moment. It also maintains a `.gitignore` entry that excludes the lockfile from version control while ensuring it is automatically tracked if absent. + +### Sentinel Error Contract + +Package-level variables: **`ErrPathTraversal`** (`error`) β€” returned when a resolved path escapes outside the repository root boundary. **`ErrLockHeld`** (`error`) β€” returned when another code-reducer process holds an exclusive file lock on the shared resource. Both variables are initialized once via `errors.New` and remain read-only throughout their scope. No mutable state, no concurrency primitives, no panic handling exists in this file. Callers detect violations by matching against these sentinels using `errors.Is(err, ErrPathTraversal)` or equivalent patterns; because the errors are unwrapped as-is (no wrapping), identity is preserved directly through the error chain. + +### Lock Acquisition and Release + +**Struct Definition**: `SimpleLock` carries: `lockPath string` (absolute path to the lockfile on disk), `file *os.File` (open file descriptor for the lockfile, or nil if not acquired), `mu sync.Mutex` (protects internal struct fields during `Unlock()`), `closed bool` (marks whether the lock has been released and the file descriptor closed). + +**`*SimpleLock.Unlock()`**: Closes the lock file descriptor and removes the lockfile in a thread-safe manner. **Idempotency contract**: Calling `Unlock()` on an already-closed lock returns `nil`. This is intentional, not accidental. Error handling inside the method: If `l.closed == true` β†’ return `nil`. Close the file descriptor (`os.File.Close`) β€” error captured in local variable; if close fails, set `file = nil` so a subsequent close becomes a no-op. Remove the lockfile via `os.Remove`. If both close and remove fail: remove error overwrites prior close error only when close returned without error. If close failed first, any remove error is ignored. This means the final returned error reflects whichever operation last produced an error, not necessarily the most severe one β€” an intentional design choice worth flagging for callers. If `os.IsNotExist(err)` on remove β†’ silently accepted as normal idempotent cleanup. + +### Path Resolution + +**Signature**: `func SafeResolve(repoRoot, inputPath string) (string, error)`. **Responsibility**: Resolve any input path strictly inside the repository root. The function performs a multi-step containment check to defeat symlink-based escape vectors and physical directory traversal. + +**Algorithmic Steps**: (1) Call `filepath.Abs` on `inputPath`. (2) Evaluate symlinks on the existing ancestor via `EvalSymlinks` β€” prevents symlink-based escape from parent directories that exist but are symbolic links. (3) Walk up from the target until a physically-existing directory is found, then re-evaluate symlinks on that ancestor via `EvalSymlinks` again. (4) Rejoin with the remaining suffix components after the physical ancestor boundary. (5) Verify final result is under `resolvedRoot` via relative path check (`filepath.Rel`). + +**Error Handling**: Any error from `Abs`, first `EvalSymlinks`, second `EvalSymlinks`, or `Lstat` (non-exist) wraps with descriptive message using `fmt.Errorf(..., err)` β€” underlying cause recovered via `%w`. Unrecognized error from `Lstat` (`!os.IsNotExist`) returns immediately as wrapped error. Path escapes repo root (`..` prefix or non-nil rel) returns sentinel-style error wrapping `ErrPathTraversal` with the original input path. + +### Lock Acquisition (Detailed) + +**Signature**: `func AcquireLock(repoRoot string) (*SimpleLock, error)`. **Responsibility**: Establish an exclusive process-level lock on a shared resource. Uses `O_EXCL` flag for atomic creation of the lockfile β€” if the lockfile already exists, acquisition fails immediately without silent success. + +**Lockfile Lifecycle**: Resolve path calls `SafeResolve(repoRoot)` to obtain absolute lock path. Create atomically opens file with `O_WRONLY | O_CREATE | O_EXCL` β€” atomic exclusive creation prevents race between existence check and open. Record identity writes PID of the acquiring process to the newly created lockfile. Failure cleanup on failure: closes file, removes lockfile. + +**Error Handling**: `SafeResolve` fails propagates as-is to caller. File open fails AND error is `os.IsExist` (lock already held) returns wrapped sentinel error wrapping `ErrLockHeld`, including the lock path β€” caller can detect stale-lock condition specifically via `errors.Is(err, ErrLockHeld)`. File open fails with any other error wraps: `"failed to acquire lock at %s: "`. Write PID fails file closed, lockfile removed (cleanup), wrapped error returned. + +### Gitignore Maintenance + +**Signature**: `func EnsureGitignoreHasLockfile(repoRoot string) (error)`. **Responsibility**: Maintain the convention that the lockfile is excluded from version control but automatically tracked if missing. Appends the lockfile entry to `.gitignore` only if not already present; uses atomic rename via temp file + sync before close to prevent partial writes. + +**Algorithmic Steps**: (1) Resolve `.gitignore` path via `SafeResolve(repoRoot)`. (2) Read existing `.gitignore` content (`os.ReadFile`). (3) Parse lines, check if lockfile entry already exists. (4) If not present: construct new content with the lockfile entry appended (with trailing-newline normalization). (5) Create a temp file in same directory as `.gitignore`. (6) Write to temp file β†’ call `Sync()` on it β†’ close it β†’ `Chmod` it. (7) Rename temp file over original `.gitignore`. + +**Error Handling**: `SafeResolve` fails propagates as-is. Read fails AND not `os.IsNotExist` (file exists but read error) wraps: `"error reading .gitignore: "` β€” caller must handle missing-file vs I/O-error distinction separately. CreateTemp fails wraps with descriptive message including `%w`. Write to temp file fails wraps; temp file is NOT cleaned up (defer handles close + remove, but write error returns before cleanup path executes). Sync fails wraps β€” note that `os.Chmod` and subsequent rename may have already run on a partially-written temp file if this error occurs late in the sequence. Close fails wraps. Chmod fails wraps. Rename (atomic swap) fails wraps with descriptive message. + +The deferred function (`tmpFile.Close()` + `os.Remove(tmpName)`) runs regardless of return path. Because the function returns before the defer has a chance to execute in some failure paths, cleanup may not happen for intermediate states β€” but since we're using temp file with unique name and same-directory rename, the OS-level temp file is cleaned by `os.Remove` in the defer on any error return path. If *write* fails mid-stream before close/rename, the deferred Remove will clean up the partially-written temp file. + +### Data Flow Summary (Security Module) + +``` +[Initialize] β†’ [Ensure path safety for all repo-root operations (SafeResolve)] β†’ +[If no process holds the lock] β†’ [Create lockfile with PID via O_EXCL (AcquireLock)] β†’ +[Run business logic within locked session] β†’ +[Append entry to .gitignore if needed (EnsureGitignoreHasLockfile)] +``` + +## Tools Subsystem + +### Module Responsibility + +The `internal/tools` package provides repository-scanning primitives for a codebase-analysis tool. It exposes two functional domains: **safe local file I/O** (`file_tools.go`) and **git subprocess orchestration** (`git_tools.go`). Together they enable the caller to discover source-code files, classify their content type, read/write artifacts safely, and validate that the target directory is a git work-tree before proceeding. + +### File Operations + +**Safe Read / Write Primitive**: `ReadFileSafely` and `WriteFileSafely` implement TOCTOU-safe file operations using the temp-file + rename pattern. Shared sequence: (1) Resolve a virtual path to an absolute filesystem path via `SafeResolve`. (2) On write: create parent directories with `defaultDirPerm`; on read: no directory creation. (3) Open or create a temporary file in the target directory using `os.CreateTemp`. (4) Write content (for writes) or read from the resolved path (for reads). (5) Close and sync (writes only); close (reads only). (6) Rename temp into final destination for writes; return contents for reads. + +**Symlink detection**: Both paths verify the target is not a symlink via `os.Lstat`/`f.Stat`. Pre-open checks prevent TOCTOU swap attacks where an attacker replaces a real file with a symlink between resolution and open. Post-open checks catch any state changes after the initial open. + +**Error semantics**: `ReadFileSafely`: returns `nil, err` on failure; wraps each step with context strings (`failed to open file`, `failed to lstat`, etc.). Symlink detection yields an explicit "security violation" error rather than a generic I/O error. `WriteFileSafely`: all failure paths return wrapped errors (directory creation, temp write, sync, close, chmod, rename). No panic usage; no sentinel values. Deferred cleanup: Both functions defer `f.Close()` for reads and use deferred `os.Remove` in `WriteFileSafely` to clean up temp files if the rename fails. + +**Gitignore Handling**: **`LoadGitignore(repoRoot)`**: Opens `.gitignore` under `repoRoot`. If absent (`os.IsNotExist`), returns `nil, nil` β€” no error raised. Any other open/read failure is returned unwrapped. **`ShouldIgnoreFile(repoRoot, relPath, gitIgnore)`**: Applies layered rules: (1) Check against compiled `GitIgnore` patterns (user-supplied + `.gitignore`). (2) Reject any path component starting with `.` or ending with `.egg-info`, regardless of other rules. (3) If extension matches a known text-source language list, accept as code. (4) For unknown extensions: resolve absolute path and scan first 1024 bytes for null bytes; binaries return `true`. + +**Side effect note**: `ShouldIgnoreFile` opens the file on every invocation to classify unknown extensions β€” this is a real I/O side effect per call, not cached. + +### Code Discovery Pipeline + +**Input**: repository root + list of ignore patterns (strings). **Output**: slice of relative source-code paths. + +**Algorithmic flow**: (1) Compile user-supplied ignore patterns and `.gitignore` entries into a single `GitIgnore` object via `LoadGitignore`. (2) Walk the entire tree recursively using `filepath.WalkDir`. For each entry: Directory: if name starts with `.` or matches any compiled pattern, skip subtree entirely. File: delegate to `ShouldIgnoreFile`. (3) Collect passing files into result slice. + +**Walk error handling**: If a walk callback returns an error for a single entry, the error is printed to `stderr` and that item is skipped (`return nil`). The overall `err` from `WalkDir` propagates at the end β€” terminal fatal errors are returned; non-fatal entries are swallowed silently. + +### Binary Classification + +**IsBinaryFile**: Scans first 1024 bytes for null bytes to distinguish text source files from binary content. Returns `true` (ignored) on any I/O error other than EOF β€” this "fail-closed" assumption treats unreadable files as likely binary rather than propagating the actual error. + +### Git Tools + +**External Command Execution (`RunGit`)**: Spawns `git --no-pager` via `exec.Command` with any additional arguments from the caller. The subprocess runs within `repoRoot` as its working directory. Stdout/stderr are captured into in-memory `bytes.Buffer`; no file, network, or DB I/O occurs. + +**Failure handling**: On error, wraps the original with stderr content: +``` +fmt.Errorf("git command failed: %v, stderr: %s", err, trimmedErr) +``` +Returns `(string, error)` β€” partial stdout is still delivered alongside the error. No panics. + +**Repository Validation (`VerifyGitRepo`)**: Delegates to `RunGit` with subcommand `rev-parse --is-inside-work-tree`. If this invocation results in an error, returns a fixed message: `"not a git repository (or any of the parent directories)"`. The original stderr is discarded at this wrapper layer β€” normalized to a single sentinel-style message for downstream callers. + +## Concurrency & State Summary + +**Mutable state**: None across all modules. All constants are `const` (compile-time); all variables (`safePath`, `dir`, `tmpFile`, `tmpName`, `patterns`, `files`, `buf`, `cmd`, `stdout`, `stderr`, etc.) are function-local and never escape their defining scope. No package-level global state is modified. + +**Concurrency mechanisms**: None detected across any module. No `sync.Mutex`, channels, `atomic` types, or other synchronization primitives. Shared mutable state does not exist to protect. The engine runner's returned error is then wrapped as `"documentation run failed: …"`. \ No newline at end of file