Bound indexing resource usage with configurable limits

[liuchong]

Problem

An accidental index_repository request against an aggregate workspace or unexpectedly large tree can consume unbounded CPU, memory, temporary storage, and database space. Discovery currently has no unified admission policy shared with supervised workers and persistence.

Proposed behavior

Resolve one immutable resource policy per ordinary indexing request and enforce it across:

• repository-root admission and bounded discovery (files, directories, entries, depth, source bytes, per-file bytes, scan deadline)
• worker execution (CPU worker count, physical concurrency, resident memory, lower priority, wall-clock duration)
• persistence (database, staging, task temporary output, cache admission, reserved free disk)
• structured resource_limit_exceeded responses that preserve the last published database

Built-in exact denied roots would include the filesystem root, current user home, and the index cache. Operators could add exact aggregate roots with index_denied_roots; descendants would remain indexable.

cross-repo-intelligence would retain its existing database-linking path because it does not walk source files.

Proposed defaults

The prototype uses conservative finite defaults: 100k files, 20k directories, 500k entries, depth 64, 4 GiB accepted source, 64 MiB per file, 30-second discovery, 4 worker threads, one physical job, 8 GiB memory (also clamped to detected budget), 16 GiB database, 20 GiB staging, 24 GiB task temp, 32 GiB cache admission, 4 GiB free-disk reserve, and one-hour duration.

Prototype

A complete C implementation with tests and documentation is available for design review:

https://github.com/liuchong/codebase-memory-mcp/tree/fix/index-resource-guards

Focused ASan/UBSan suites: 640 passed, 3 platform skips. make -f Makefile.cbm lint-ci passes.

Design questions

1. Is a unified index_* configuration namespace acceptable?
2. Are the proposed built-in denied roots and defaults appropriate?
3. Should the implementation be reviewed as one cross-cutting PR, or split along discovery/supervision/persistence boundaries despite the shared policy contract?

[DeusData]

Thank you for proposing a unified resource envelope and for following it with PR #1348. This is a useful direction, but defaults, enforcement points, cancellation behavior, and compatibility for unconstrained existing installs are project-level policy decisions rather than mechanical limits. I have routed the proposal to 0.9.2-rc for that design review, without committing to the current PR shape yet.

[liuchong]

Thanks, noted. Happy to wait on the design review call for defaults and compatibility before reshaping #1348. 👀