Skip to content

fix sigma modifier chains and true multi-engine service-map p95#261

Merged
Polliog merged 4 commits into
developfrom
fix/sigma-modifiers-true-p95-255
Jun 22, 2026
Merged

fix sigma modifier chains and true multi-engine service-map p95#261
Polliog merged 4 commits into
developfrom
fix/sigma-modifiers-true-p95-255

Conversation

@Polliog

@Polliog Polliog commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

Closes #255. Two correctness follow-ups from the bug-hunt sweep, both validated against real ClickHouse, MongoDB and TimescaleDB.

Item 1 - Sigma compound field-modifier chains

parseFieldWithModifier split a field key on | but kept only the first modifier, so chains like CommandLine|utf16le|base64offset|contains (PowerShell -enc style) matched incorrectly.

  • Whole chain is now parsed and applied in order: transforms rewrite the pattern, then the final comparator runs.
  • Transforms: base64, base64offset, utf16le/utf16/utf16be/wide, windash. Comparators: existing + new cidr and numeric gt/gte/lt/lte.
  • Two deliberate SigmaHQ-spec corrections (real synced rules are authored against these): base64/base64offset now ENCODE the pattern (field checked for base64(value)) rather than decoding the field; |all is now the list quantifier (OR -> AND over a value list), not the old "all whitespace-split words in any order". The two old tests asserting the previous behavior were replaced.

Item 2 - true service-map p95 (multi-engine)

The map reported MAX(duration_p95_ms) from the per-bucket spans continuous aggregate, which overestimates (a p95 is not derivable from per-bucket p95s) and was only ever produced on TimescaleDB.

  • The t-digest-sketch approach from the issue was rejected: the prod/test image timescale/timescaledb:latest-pg16 lacks timescaledb_toolkit, and MongoDB has no mergeable sketch.
  • New reservoir.getServiceHealthStats computes a true window p95 directly from raw spans per engine: percentile_cont (TimescaleDB), quantile(0.95) (ClickHouse), $percentile (MongoDB, approximate, 7.0+). ClickHouse/Mongo service maps now carry real call/error/latency/p95 figures where they previously had none.
  • traces/service.ts calls the new method; spans_hourly_stats/spans_daily_stats are unchanged and still back the dashboards.

No database migrations.

Tests

  • field-matcher 43/43, sigma backend dir 217/217, condition-evaluator + detection-engine green
  • reservoir engine suites 191/191 incl. real ClickHouse + MongoDB; new getServiceHealthStats integration tests per engine
  • traces service 42/42 on real TimescaleDB (rewrote the obsolete "defaults when health stats empty" case to assert the true window p95)
  • backend typecheck + reservoir build clean

@codecov

codecov Bot commented Jun 22, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.12346% with 32 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...ackages/backend/src/modules/sigma/field-matcher.ts 87.92% 25 Missing ⚠️
packages/reservoir/src/client.ts 12.50% 7 Missing ⚠️

📢 Thoughts on this report? Let us know!

@Polliog Polliog merged commit c6e19c8 into develop Jun 22, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant