Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,14 @@
Preserve Intel macOS builds by excluding the unavailable ONNX runtime on that
target; native document processing remains available and managed OCR reports
an explicit unsupported-platform error before downloading model weights.
The graph viewer now exposes combined/native/OCR evidence tabs, confidence
and page/slide/sheet locators, bounded OCR region previews, coverage warnings,
and a managed model status/install card during initialization.

- Make semantic provider choice explicit and repeatable with `--backend`,
`--model`, `COMPASS_BACKEND`, and `COMPASS_MODEL`. Credential guidance now
covers all built-in providers and custom provider environment keys without
exposing secrets in Compass configuration or artifacts.

- Hard-cut Swift, Dart, Scala, and Groovy/Gradle onto version-1 qualifying
universal evidence pipelines. The bounded AST-first producer publishes
Expand Down
11 changes: 11 additions & 0 deletions COMPATIBILITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,12 @@ with exact source owner, geometry, confidence, and model provenance. Partial
visual coverage is never labeled complete or finalized as a complete document
cache entry.

Semantic enrichment accepts explicit provider/model selection through
`--backend`/`--model` or the non-secret `COMPASS_BACKEND`/`COMPASS_MODEL`
environment defaults. Provider credentials remain provider-specific
environment or secret-store values and are excluded from graph artifacts,
history profiles, and cache identities.

## Evolving contracts

The Grounded Agent Graph feature is additive and opt-in. It does not change
Expand Down Expand Up @@ -193,6 +199,11 @@ Standalone HTML may additionally embed optional, presentation-only source
navigation metadata for a recognized Git forge and full source commit. This
metadata is outside `compass.viewer.workbench/1`; `workbench-json` and the
versioned graph/view contracts are unchanged.
Rich document nodes may carry an optional additive `document` object in
`compass.viewer.graph/1`. It contains bounded native/OCR provenance, typed
page/slide/sheet locators, confidence, and OCR geometry for the viewer; older
readers may ignore the field, while strict readers should preserve unknown
nested fields and fail closed only on an unknown contract major.
Structural builds publish a validated `store.sqlite3` sidecar and typed
`store.ref` selector by default; `--store json` explicitly opts out. Typed code
queries prefer that validated sidecar by default, while `--engine json`
Expand Down
6 changes: 3 additions & 3 deletions crates/compass-cli/src/help.rs
Original file line number Diff line number Diff line change
Expand Up @@ -749,9 +749,9 @@ const PAGES: &[Page] = &[
),
page!(
"provider",
"Manage custom semantic model providers",
"Choose built-in or custom semantic model providers",
["compass provider <COMMAND>"],
"Examples:\n compass provider list\n compass provider show local\n\nTips:\n Run `compass help provider add` for endpoint and credential configuration."
"Examples:\n compass provider list\n compass provider show local\n COMPASS_BACKEND=gemini GEMINI_API_KEY=… compass extract ./docs\n\nBuilt-in providers include claude, kimi, ollama, gemini, openai, deepseek, azure, bedrock, and claude-cli. Use `--backend <NAME>` (or `COMPASS_BACKEND`) when more than one credential is available.\n\nTips:\n Run `compass help provider add` for custom OpenAI-compatible endpoints and credential configuration. Keys are read from environment variables only; they are not stored in `.compass/config.toml`, provider registries, history profiles, or graph artifacts."
),
page!(
"provider add",
Expand All @@ -765,7 +765,7 @@ const PAGES: &[Page] = &[
"provider list",
"List registered custom providers",
["compass provider list"],
"Examples:\n compass provider list"
"Examples:\n compass provider list\n\nNotes:\n This lists custom providers registered in `~/.compass/providers.json`. Built-in providers are selected with `--backend` or `COMPASS_BACKEND`; see `compass help extract` for the supported credential variables."
),
page!(
"provider show",
Expand Down
52 changes: 50 additions & 2 deletions crates/compass-cli/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2097,6 +2097,13 @@ fn command_build_with_validation_inner(
"error: must specify a path to scan or a --postgres DSN".to_owned(),
);
}
if extract {
// Explicit flags win over environment configuration. The generic
// selector is intentionally non-secret: provider transports still
// read only their documented provider-specific credential variable.
let environment = std::env::vars().collect::<HashMap<_, _>>();
apply_provider_environment_defaults(&mut backend, &mut model, &environment);
}
let root = if extract && !has_explicit_root {
PathBuf::from(".")
} else {
Expand Down Expand Up @@ -2508,11 +2515,32 @@ fn no_llm_api_key_message(semantic_count: usize, dedup_llm: bool) -> String {
""
};
format!(
"no LLM API key found ({}). Set GEMINI_API_KEY or GOOGLE_API_KEY (gemini), MOONSHOT_API_KEY (kimi), ANTHROPIC_API_KEY (claude), OPENAI_API_KEY (openai), DEEPSEEK_API_KEY (deepseek), or pass --backend. A code-only corpus needs no key.{hint}",
"no LLM API key found ({}). Choose a provider with --backend <NAME> or COMPASS_BACKEND and set its credential: GEMINI_API_KEY/GOOGLE_API_KEY (gemini), MOONSHOT_API_KEY (kimi), ANTHROPIC_API_KEY (claude), OPENAI_API_KEY (openai), DEEPSEEK_API_KEY (deepseek), AZURE_OPENAI_API_KEY plus AZURE_OPENAI_ENDPOINT (azure), AWS credentials (bedrock), OLLAMA_BASE_URL for local Ollama, or the env key registered with `compass provider add`. A code-only corpus needs no key.{hint}",
reasons.join("; ")
)
}

fn apply_provider_environment_defaults(
backend: &mut Option<String>,
model: &mut Option<String>,
environment: &HashMap<String, String>,
) {
if backend.is_none() {
*backend = environment_value(environment, "COMPASS_BACKEND");
}
if model.is_none() {
*model = environment_value(environment, "COMPASS_MODEL");
}
}

fn environment_value(environment: &HashMap<String, String>, key: &str) -> Option<String> {
environment
.get(key)
.map(|value| value.trim())
.filter(|value| !value.is_empty())
.map(str::to_owned)
}

fn command_hook_refresh(frontend: Frontend, args: &[String]) -> Outcome {
let launch_root = args
.iter()
Expand Down Expand Up @@ -3002,7 +3030,7 @@ fn executable_on_path(name: &str) -> bool {
}

fn extract_help() -> String {
"Usage: compass extract [PATH] [--program] [--program-artifact PATH] [--no-program] [--store json|sqlite] [--inference-level low|medium|high|max] [--code-only] [--cargo] [--google-workspace] [--postgres DSN] [--backend NAME] [--model MODEL] [--mode deep] [--ocr off|auto|always] [--ocr-profile NAME] [--ocr-language BCP47] [--token-budget N] [--max-concurrency N] [--max-workers N] [--max-source-bytes N] [--api-timeout SECONDS] [--allow-partial] [--dedup-llm] [--timing] [--out DIR] [--no-cluster] [--force] [--no-viz] [--no-gitignore] [--exclude PATTERN] [--resolution N] [--exclude-hubs N]".to_owned()
"Usage: compass extract [PATH] [--program] [--program-artifact PATH] [--no-program] [--store json|sqlite] [--inference-level low|medium|high|max] [--code-only] [--cargo] [--google-workspace] [--postgres DSN] [--backend NAME] [--model MODEL] [--mode deep] [--ocr off|auto|always] [--ocr-profile NAME] [--ocr-language BCP47] [--token-budget N] [--max-concurrency N] [--max-workers N] [--max-source-bytes N] [--api-timeout SECONDS] [--allow-partial] [--dedup-llm] [--timing] [--out DIR] [--no-cluster] [--force] [--no-viz] [--no-gitignore] [--exclude PATTERN] [--resolution N] [--exclude-hubs N]\nProvider selection: --backend/--model override COMPASS_BACKEND/COMPASS_MODEL. Built-ins: claude, kimi, ollama, gemini, openai, deepseek, azure, bedrock, claude-cli. Set the selected provider's documented credential variable; custom providers use `compass provider add`. Credentials are never written to Compass artifacts.".to_owned()
}

fn saved_graph_root() -> Option<PathBuf> {
Expand Down Expand Up @@ -6580,4 +6608,24 @@ mod mcp_option_tests {
assert!(extract_help().contains("--inference-level low|medium|high|max"));
Ok(())
}

#[test]
fn provider_environment_defaults_are_non_secret_and_flags_win() {
let environment = HashMap::from([
("COMPASS_BACKEND".to_owned(), " gemini ".to_owned()),
("COMPASS_MODEL".to_owned(), " gemini-test ".to_owned()),
]);
let mut backend = None;
let mut model = None;
apply_provider_environment_defaults(&mut backend, &mut model, &environment);
assert_eq!(backend.as_deref(), Some("gemini"));
assert_eq!(model.as_deref(), Some("gemini-test"));

let mut backend = Some("openai".to_owned());
let mut model = Some("gpt-test".to_owned());
apply_provider_environment_defaults(&mut backend, &mut model, &environment);
assert_eq!(backend.as_deref(), Some("openai"));
assert_eq!(model.as_deref(), Some("gpt-test"));
assert!(extract_help().contains("COMPASS_BACKEND/COMPASS_MODEL"));
}
}
4 changes: 3 additions & 1 deletion crates/compass-cli/tests/dedup_extract.rs
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ const BACKEND_ENVIRONMENT: &[&str] = &[
"AWS_DEFAULT_REGION",
"AWS_ACCESS_KEY_ID",
"OLLAMA_BASE_URL",
"COMPASS_BACKEND",
"COMPASS_MODEL",
];

#[test]
Expand All @@ -49,7 +51,7 @@ fn dedup_llm_without_backend_has_actionable_diagnostic() -> Result<(), Box<dyn E
assert!(output.stdout.is_empty());
let stderr = String::from_utf8(output.stderr)?;
assert!(stderr.starts_with(
"error: no LLM API key found (--dedup-llm was passed). Set GEMINI_API_KEY or GOOGLE_API_KEY (gemini), MOONSHOT_API_KEY (kimi), ANTHROPIC_API_KEY (claude), OPENAI_API_KEY (openai), DEEPSEEK_API_KEY (deepseek), or pass --backend. A code-only corpus needs no key.\n"
"error: no LLM API key found (--dedup-llm was passed). Choose a provider with --backend <NAME> or COMPASS_BACKEND and set its credential: GEMINI_API_KEY/GOOGLE_API_KEY (gemini), MOONSHOT_API_KEY (kimi), ANTHROPIC_API_KEY (claude), OPENAI_API_KEY (openai), DEEPSEEK_API_KEY (deepseek), AZURE_OPENAI_API_KEY plus AZURE_OPENAI_ENDPOINT (azure), AWS credentials (bedrock), OLLAMA_BASE_URL for local Ollama, or the env key registered with `compass provider add`. A code-only corpus needs no key.\n"
));
assert!(stderr.contains("Compass extract failed after "));
Ok(())
Expand Down
12 changes: 12 additions & 0 deletions crates/compass-core/src/document_processing.rs
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ pub struct PreparedDocument {
pub artifact: Arc<DocumentArtifact>,
pub semantic_text: Arc<str>,
pub cache_identity: String,
pub ocr_mode: OcrMode,
}

#[derive(Clone, Debug)]
Expand Down Expand Up @@ -152,6 +153,7 @@ pub fn prepare_document_set(
artifact: Arc::new(artifact),
semantic_text: Arc::<str>::from(semantic_text),
cache_identity: cache_identity.clone(),
ocr_mode: options.ocr_mode,
},
);
}
Expand Down Expand Up @@ -252,6 +254,7 @@ pub(crate) fn project_document(
path: &Path,
artifact: &DocumentArtifact,
cache_identity: &str,
ocr_mode: OcrMode,
) -> Result<compass_languages::Extraction, String> {
artifact.validate().map_err(|error| error.to_string())?;
let file_id = compass_languages::make_id(&[source_file]);
Expand Down Expand Up @@ -294,6 +297,10 @@ pub(crate) fn project_document(
"document_visual_coverage".to_owned(),
serde_json::to_value(artifact.visual_coverage).map_err(|error| error.to_string())?,
);
root.insert(
"document_ocr_mode".to_owned(),
serde_json::to_value(ocr_mode).map_err(|error| error.to_string())?,
);
if !artifact.metadata.is_empty() {
root.insert(
"document_metadata".to_owned(),
Expand Down Expand Up @@ -486,6 +493,7 @@ mod tests {
Path::new("report.docx"),
&artifact,
"fixture-identity",
OcrMode::Off,
)?;
assert_eq!(extraction.nodes.len(), 2);
assert_eq!(extraction.edges.len(), 1);
Expand All @@ -496,6 +504,10 @@ mod tests {
.and_then(serde_json::Value::as_str),
Some("Projected sentinel")
);
assert_eq!(
extraction.nodes[0].attributes.get("document_ocr_mode"),
Some(&serde_json::json!("off"))
);
assert_eq!(
extraction.extensions.get("document_cache_identity"),
Some(&serde_json::json!("fixture-identity"))
Expand Down
1 change: 1 addition & 0 deletions crates/compass-core/src/pipeline.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2907,6 +2907,7 @@ fn build_graph_inner_unscoped(
path,
&prepared.artifact,
&prepared.cache_identity,
prepared.ocr_mode,
)
.map_err(|detail| {
compass_languages::ExtractError::InvalidDocumentEvidence {
Expand Down
62 changes: 31 additions & 31 deletions crates/compass-output/assets/viewer/graph.js

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions crates/compass-output/assets/viewer/manifest.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@
"viewerSchema": "compass.viewer.graph/1",
"files": {
"graph.js": {
"bytes": 1086882,
"sha256": "c1839aa0199937143cfdf39c3876c9f3eeb15d9fded754fc7863bf9982f5316c"
"bytes": 1098604,
"sha256": "40635538ce94903b438ae439d15c0609f384e304db3d300e93c5859637878263"
},
"viewer.css": {
"bytes": 255848,
"sha256": "a9b91d60549e949999805c6791b3f99c34e3788efb9513757458c4ea673a8646"
"bytes": 272342,
"sha256": "6aafbe98455900b02652aa406f566e2dcc9293fc30a5db991342ba408d079279"
}
}
}
2 changes: 1 addition & 1 deletion crates/compass-output/assets/viewer/viewer.css

Large diffs are not rendered by default.

104 changes: 104 additions & 0 deletions crates/compass-output/src/html.rs
Original file line number Diff line number Diff line change
Expand Up @@ -689,6 +689,9 @@ pub(crate) fn node_values(
);
output.insert("end_byte".into(), end_byte.map_or(Value::Null, Value::from));
output.insert("signature".into(), Value::String(signature.clone()));
if let Some(document) = document_value(node) {
output.insert("document".into(), document);
}
if let Some(counts) = options.member_counts {
output.insert("is_community".into(), Value::Bool(true));
output.insert(
Expand Down Expand Up @@ -738,6 +741,107 @@ fn source_anchor_unsigned(node: &NodeRecord, legacy_key: &str, v1_key: &str) ->
})
}

fn document_value(node: &NodeRecord) -> Option<Value> {
const DOCUMENT_VIEWER_TEXT_LIMIT: usize = 4_096;
let is_document =
node.string("file_type") == "document" || node.property("document_kind").is_some();
if !is_document {
return None;
}
let kind = if node.string("document_kind").is_empty() {
node.string("symbol_kind")
} else {
node.string("document_kind")
};
let role = if kind == "document" { "root" } else { "block" };
let mut document = Map::new();
document.insert("role".into(), Value::String(role.to_owned()));
if !kind.is_empty() {
document.insert("kind".into(), Value::String(kind));
}
for (target, source) in [
("format", "document_format"),
("visualCoverage", "document_visual_coverage"),
("ocrMode", "document_ocr_mode"),
] {
if let Some(value) = node
.property(source)
.and_then(|value| bounded_document_json(&value, 0))
{
document.insert(target.to_owned(), value);
}
}
if let Some(text) = node
.property("document_text")
.and_then(|value| value.as_str().map(str::to_owned))
{
document.insert(
"text".to_owned(),
Value::String(sanitize_metadata(&text, DOCUMENT_VIEWER_TEXT_LIMIT)),
);
}
if let Some(value) = node
.property("document_complete")
.and_then(|value| bounded_document_json(&value, 0))
{
document.insert("complete".into(), value);
}
if let Some(value) = node
.property("block_index")
.and_then(|value| bounded_document_json(&value, 0))
{
document.insert("ordinal".into(), value);
}
for (target, source) in [
("origin", "document_origin"),
("locator", "document_locator"),
("ocrProfile", "document_ocr_profile"),
] {
if let Some(value) = node
.property(source)
.and_then(|value| bounded_document_json(&value, 0))
{
document.insert(target.to_owned(), value);
}
}
Some(Value::Object(document))
}

fn bounded_document_json(value: &Value, depth: usize) -> Option<Value> {
const MAX_DEPTH: usize = 5;
const MAX_OBJECT_ENTRIES: usize = 128;
const MAX_ARRAY_ENTRIES: usize = 512;
const MAX_STRING_CHARS: usize = 16_384;

if depth > MAX_DEPTH {
return None;
}
match value {
Value::Null => None,
Value::Bool(_) | Value::Number(_) => Some(value.clone()),
Value::String(value) => Some(Value::String(sanitize_metadata(value, MAX_STRING_CHARS))),
Value::Array(values) => Some(Value::Array(
values
.iter()
.take(MAX_ARRAY_ENTRIES)
.filter_map(|value| bounded_document_json(value, depth + 1))
.collect(),
)),
Value::Object(values) => Some(Value::Object(
values
.iter()
.take(MAX_OBJECT_ENTRIES)
.filter_map(|(key, value)| {
Some((
sanitize_metadata(key, 128),
bounded_document_json(value, depth + 1)?,
))
})
.collect(),
)),
}
}

#[allow(dead_code)]
fn community_details(
document: &GraphDocument,
Expand Down
7 changes: 4 additions & 3 deletions crates/compass-output/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -80,9 +80,10 @@ pub use review::{
pub use svg::{SvgOptions, spring_layout, svg_document, write_svg};
pub use tree::{TreeNode, TreeOptions, build_tree, tree_html_document, write_tree_html};
pub use viewer_model::{
EffectiveGraphViewContext, GRAPH_VIEWER_SCHEMA, GraphViewCommunity, GraphViewEdge,
GraphViewModel, GraphViewNode, GraphViewSource, GraphViewStats, effective_graph_view_model,
graph_view_model, shared_viewer_html, shared_viewer_html_with_communities,
EffectiveGraphViewContext, GRAPH_VIEWER_SCHEMA, GraphViewCommunity, GraphViewDocument,
GraphViewEdge, GraphViewModel, GraphViewNode, GraphViewSource, GraphViewStats,
effective_graph_view_model, graph_view_model, shared_viewer_html,
shared_viewer_html_with_communities,
};
pub use wiki::{WikiExport, WikiOptions, export_wiki};
pub use workbench::{
Expand Down
Loading
Loading