feat: enhance cost estimation fit logic

[AnkitKmrGupta]

This pull request introduces significant schema and ingestion changes to support segmented cost model fitting and improved fan detection in the cost analytics pipeline. The main changes include adding new columns to the cost_daily_stats and cost_fee_model tables, updating the ingestion logic for Adyen to extract both interchange category and rate, and expanding the data structures and queries to handle segmentation and new error metrics. These updates enable more granular and accurate modeling, especially for cases with overlapping interchange rates and segmented fits.

Schema and Data Model Changes:

• clickhouse/scripts/035_cost_model.sh: Added new columns to cost_daily_stats and cost_fee_model for segmentation (fit_bucket, segment_idx, amount_lo, amount_hi), interchange rate (interchange_bps), and detailed error metrics (grade_bps, pct_ci95_bps, astar, prop_bps, fix_abs, fix_bps, below_gross_frac, fan_frac, fan_money_bps). Updated table sorting keys and included migration statements for existing deployments. [1] [2] [3] [4]

Data Ingestion and Extraction:

• src/cost_ingestion/connectors/adyen.rs: Refactored to extract both interchange category and rate from Adyen settlement reports, including normalization of the rate. Updated tests to cover new extraction logic. [1] [2] [3] [4]
• src/cost_ingestion/connectors/braintree.rs, src/cost_ingestion/connectors/chase.rs, src/cost_ingestion/connectors/checkout.rs, src/cost_ingestion/connectors/stripe.rs: Updated ingestion logic to include the new interchange_bps field (set to empty string where not available). [1] [2] [3] [4]

Cost Model Aggregation and Querying:

• src/cost_ingestion/blended.rs: Updated TopCluster and related aggregation logic to handle new segmentation and error metric fields. The SQL query and result parsing now include all new fields, and grouping is extended to support segmentation and overlapping interchange rates. [1] [2] [3] [4]

These changes lay the groundwork for more granular cost modeling and improved detection of non-linearities and "fan" effects in the transaction data.

[Copilot]

Pull request overview

This PR upgrades the cost analytics pipeline to support segmented model fitting, interchange-rate-aware clustering, and “fan” (mixed-population) detection, with corresponding schema changes, ingestion updates, serving lookup adjustments, and dashboard/UI updates.

Changes:

• Extends ClickHouse cost_daily_stats / cost_fee_model schema to store fit segmentation keys, interchange rate, bounded samples, and richer error metrics (including FAN verdict support).
• Updates ingestion/rollup to populate interchange_bps, fit_bucket, and reservoir samples, and rewrites fitting logic to use segmentation + fan detection while persisting expanded metrics.
• Updates serving, overrides, APIs, and website UI/types to handle segmented clusters and new coverage/verdict fields.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
clickhouse/scripts/035_cost_model.sh	Adds new columns/keys and migration ALTERs for segmentation, IC rate, sampling, and new metrics/verdicts.
src/cost_ingestion/connectors/adyen.rs	Extracts both interchange category and normalized interchange bps from ICSF details.
src/cost_ingestion/connectors/braintree.rs	Populates new interchange_bps field (empty where unavailable).
src/cost_ingestion/connectors/chase.rs	Populates new interchange_bps field (empty where unavailable).
src/cost_ingestion/connectors/checkout.rs	Populates new interchange_bps field (empty where unavailable).
src/cost_ingestion/connectors/stripe.rs	Populates new interchange_bps field (empty where unavailable).
src/cost_ingestion/types.rs	Adds interchange_bps to settled rows and introduces fit_bucket() helper.
src/cost_ingestion/rollup.rs	Changes bucket identity to include fit_bucket/interchange_bps, retains micro-txns, and adds per-bucket reservoir sampling.
src/cost_ingestion/sink.rs	Updates ClickHouse insert columns/body to include interchange_bps, fit_bucket, and sample arrays.
src/cost_ingestion/fit.rs	Replaces CH-only fit with Rust-side richer grading/segmentation/fan logic and inserts expanded model rows.
src/cost_ingestion/serving.rs	Serves segmented models, predicts (category, rate) pairs, and probes legacy/segmented override keys.
src/cost_ingestion/overrides.rs	Makes override key parsing/back-compat aware of segmented and IC-rate-aware key forms.
src/cost_ingestion/blended.rs	Expands top cluster aggregation/query parsing to include segmentation and new metrics.
src/cost_ingestion/detect.rs	Extends price-change detection to partition by (category, rate, segment).
src/cost_ingestion/coverage.rs	Adds FAN coverage counts/volume to coverage summary query and parsing.
src/routes/cost_clusters.rs	Extends cluster API payload with segmentation, IC rate, and richer quality metrics; uses shared key formatting.
src/decider/gatewaydecider/multi_objective/mod.rs	Extends CostModel to carry segmentation and IC rate fields.
src/decider/gatewaydecider/multi_objective/hypersense_client.rs	Propagates segmentation/IC rate fields through Hypersense/inhouse model flows.
src/decider/gatewaydecider/multi_objective/seed_costs.rs	Initializes new segmentation/IC rate fields in seed cost matches.
website/src/hooks/useCostRouting.ts	Extends frontend types to include FAN coverage and segmentation/IC rate fields.
website/src/components/pages/CostCoverageCard.tsx	Adds FAN verdict row to the coverage verdict table UI.
website/src/components/pages/ClustersPanel.tsx	Improves cluster labeling and displays IC rate / segment info; refactors table row rendering and editor placement.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[jagan-jaya]

we need to add a new migration instead of editing existing one