Introduce new app config

workspaces/scorecard/.changeset/metric-level-thresholds-config.md

@@ -0,0 +1,54 @@
+---
+'@red-hat-developer-hub/backstage-plugin-scorecard-backend': major
+'@red-hat-developer-hub/backstage-plugin-scorecard-backend-module-github': major
+'@red-hat-developer-hub/backstage-plugin-scorecard-backend-module-jira': major
+'@red-hat-developer-hub/backstage-plugin-scorecard-backend-module-filecheck': major
+'@red-hat-developer-hub/backstage-plugin-scorecard-backend-module-sonarqube': major
+'@red-hat-developer-hub/backstage-plugin-scorecard-backend-module-dependabot': major
+'@red-hat-developer-hub/backstage-plugin-scorecard-backend-module-openssf': major
+'@red-hat-developer-hub/backstage-plugin-scorecard-common': major
+---
+
+**BREAKING**: Scorecard provider configuration now lives under top-level `scorecard.metricProviders` instead of `scorecard.plugins`. Provider IDs must be `<datasource>.<providerName>` (no longer equal to the datasource alone). Entity annotations for thresholds use now the full metric ID instead of provider ID.
+
+Thresholds from configuration are determined by the most specific setting (**metric > provider**):
+
+1. `metricProviders.<datasource>.<providerName>.metrics.<metricName>.thresholds`
+2. `metricProviders.<datasource>.<providerName>.thresholds`
+
+Config keys are local names (no datasource prefix). Entity annotations use the full metric ID:
+`scorecard.io/<metricId>.thresholds.rules.<key>`.
+
+Filecheck provider ID is now `filecheck.fileExistence`; files move under `options`:
+
+```diff
+ scorecard:
+-  plugins:
+-    filecheck:
+-      files:
+-        license: LICENSE
+-        codeowners: CODEOWNERS
+-      thresholds: ...
+-      schedule: ...
++  metricProviders:
++    filecheck:
++      fileExistence:
++        options:
++          files:
++            license: LICENSE
++            codeowners: CODEOWNERS
++        thresholds: ...
++        schedule: ...
+```
+
+Migration from the previous `scorecard.plugins` layout:
+
+```diff
+ scorecard:
+-  plugins:
++  metricProviders:
+     github:
+       openPRs:
+         schedule: ...
+         thresholds: ...
+```

workspaces/scorecard/AGENTS.md

@@ -25,41 +25,41 @@ snake_case in the API layer only -- they are not metric IDs.
 
 ### GitHub (1 metric)
 
-| Metric ID | Type | Source |
-|---|---|---|
+| Metric ID        | Type   | Source                     |
+| ---------------- | ------ | -------------------------- |
 | `github.openPRs` | number | `GithubOpenPRsProvider.ts` |
 
 ### Jira (1 metric)
 
-| Metric ID | Type | Source |
-|---|---|---|
+| Metric ID         | Type   | Source                      |
+| ----------------- | ------ | --------------------------- |
 | `jira.openIssues` | number | `JiraOpenIssuesProvider.ts` |
 
 ### Dependabot (4 metrics)
 
-| Metric ID | Type | Source |
-|---|---|---|
+| Metric ID                   | Type   | Source                |
+| --------------------------- | ------ | --------------------- |
 | `dependabot.alertsCritical` | number | `DependabotConfig.ts` |
-| `dependabot.alertsHigh` | number | `DependabotConfig.ts` |
-| `dependabot.alertsMedium` | number | `DependabotConfig.ts` |
-| `dependabot.alertsLow` | number | `DependabotConfig.ts` |
+| `dependabot.alertsHigh`     | number | `DependabotConfig.ts` |
+| `dependabot.alertsMedium`   | number | `DependabotConfig.ts` |
+| `dependabot.alertsLow`      | number | `DependabotConfig.ts` |
 
 ### SonarQube (12 metrics)
 
-| Metric ID | Type | SonarQube API Key | Source |
-|---|---|---|---|
-| `sonarqube.qualityGate` | boolean | _(quality gate API)_ | `SonarQubeConfig.ts` |
-| `sonarqube.openIssues` | number | _(open issues API)_ | `SonarQubeConfig.ts` |
-| `sonarqube.securityRating` | number | `security_rating` | `SonarQubeConfig.ts` |
-| `sonarqube.securityIssues` | number | `vulnerabilities` | `SonarQubeConfig.ts` |
-| `sonarqube.securityReviewRating` | number | `security_review_rating` | `SonarQubeConfig.ts` |
-| `sonarqube.securityHotspots` | number | `security_hotspots` | `SonarQubeConfig.ts` |
-| `sonarqube.reliabilityRating` | number | `reliability_rating` | `SonarQubeConfig.ts` |
-| `sonarqube.reliabilityIssues` | number | `bugs` | `SonarQubeConfig.ts` |
-| `sonarqube.maintainabilityRating` | number | `sqale_rating` | `SonarQubeConfig.ts` |
-| `sonarqube.maintainabilityIssues` | number | `code_smells` | `SonarQubeConfig.ts` |
-| `sonarqube.codeCoverage` | number | `coverage` | `SonarQubeConfig.ts` |
-| `sonarqube.codeDuplications` | number | `duplicated_lines_density` | `SonarQubeConfig.ts` |
+| Metric ID                         | Type    | SonarQube API Key          | Source               |
+| --------------------------------- | ------- | -------------------------- | -------------------- |
+| `sonarqube.qualityGate`           | boolean | _(quality gate API)_       | `SonarQubeConfig.ts` |
+| `sonarqube.openIssues`            | number  | _(open issues API)_        | `SonarQubeConfig.ts` |
+| `sonarqube.securityRating`        | number  | `security_rating`          | `SonarQubeConfig.ts` |
+| `sonarqube.securityIssues`        | number  | `vulnerabilities`          | `SonarQubeConfig.ts` |
+| `sonarqube.securityReviewRating`  | number  | `security_review_rating`   | `SonarQubeConfig.ts` |
+| `sonarqube.securityHotspots`      | number  | `security_hotspots`        | `SonarQubeConfig.ts` |
+| `sonarqube.reliabilityRating`     | number  | `reliability_rating`       | `SonarQubeConfig.ts` |
+| `sonarqube.reliabilityIssues`     | number  | `bugs`                     | `SonarQubeConfig.ts` |
+| `sonarqube.maintainabilityRating` | number  | `sqale_rating`             | `SonarQubeConfig.ts` |
+| `sonarqube.maintainabilityIssues` | number  | `code_smells`              | `SonarQubeConfig.ts` |
+| `sonarqube.codeCoverage`          | number  | `coverage`                 | `SonarQubeConfig.ts` |
+| `sonarqube.codeDuplications`      | number  | `duplicated_lines_density` | `SonarQubeConfig.ts` |
 
 ### OpenSSF (18 metrics)
 
@@ -74,26 +74,26 @@ conversion lowercases first, these become `cii`, `ci`, and `sast` --
 not `cII`, `cI`, or `SAST`. Any change to the ID generation regex
 must be verified against this full table.
 
-| Check Name | Metric ID | Edge Case? |
-|---|---|---|
-| `Binary-Artifacts` | `openssf.binaryArtifacts` | |
-| `Branch-Protection` | `openssf.branchProtection` | |
-| `CII-Best-Practices` | `openssf.ciiBestPractices` | Yes -- `CII` becomes `cii` |
-| `CI-Tests` | `openssf.ciTests` | Yes -- `CI` becomes `ci` |
-| `Code-Review` | `openssf.codeReview` | |
-| `Contributors` | `openssf.contributors` | |
-| `Dangerous-Workflow` | `openssf.dangerousWorkflow` | |
-| `Dependency-Update-Tool` | `openssf.dependencyUpdateTool` | |
-| `Fuzzing` | `openssf.fuzzing` | |
-| `License` | `openssf.license` | |
-| `Maintained` | `openssf.maintained` | |
-| `Packaging` | `openssf.packaging` | |
-| `Pinned-Dependencies` | `openssf.pinnedDependencies` | |
-| `SAST` | `openssf.sast` | Yes -- all-caps becomes all-lowercase |
-| `Security-Policy` | `openssf.securityPolicy` | |
-| `Signed-Releases` | `openssf.signedReleases` | |
-| `Token-Permissions` | `openssf.tokenPermissions` | |
-| `Vulnerabilities` | `openssf.vulnerabilities` | |
+| Check Name               | Metric ID                      | Edge Case?                            |
+| ------------------------ | ------------------------------ | ------------------------------------- |
+| `Binary-Artifacts`       | `openssf.binaryArtifacts`      |                                       |
+| `Branch-Protection`      | `openssf.branchProtection`     |                                       |
+| `CII-Best-Practices`     | `openssf.ciiBestPractices`     | Yes -- `CII` becomes `cii`            |
+| `CI-Tests`               | `openssf.ciTests`              | Yes -- `CI` becomes `ci`              |
+| `Code-Review`            | `openssf.codeReview`           |                                       |
+| `Contributors`           | `openssf.contributors`         |                                       |
+| `Dangerous-Workflow`     | `openssf.dangerousWorkflow`    |                                       |
+| `Dependency-Update-Tool` | `openssf.dependencyUpdateTool` |                                       |
+| `Fuzzing`                | `openssf.fuzzing`              |                                       |
+| `License`                | `openssf.license`              |                                       |
+| `Maintained`             | `openssf.maintained`           |                                       |
+| `Packaging`              | `openssf.packaging`            |                                       |
+| `Pinned-Dependencies`    | `openssf.pinnedDependencies`   |                                       |
+| `SAST`                   | `openssf.sast`                 | Yes -- all-caps becomes all-lowercase |
+| `Security-Policy`        | `openssf.securityPolicy`       |                                       |
+| `Signed-Releases`        | `openssf.signedReleases`       |                                       |
+| `Token-Permissions`      | `openssf.tokenPermissions`     |                                       |
+| `Vulnerabilities`        | `openssf.vulnerabilities`      |                                       |
 
 ### Filecheck (dynamic)
 
@@ -147,64 +147,56 @@ or all `"boolean"`).
 ### Threshold resolution priority
 
 `ThresholdResolver` (in `scorecard-backend`) resolves thresholds for
-each metric using a three-tier override chain. Higher tiers override
-lower tiers:
+each metric, stored per **metric ID** in `ThresholdResolver.configuredThresholds`.
+Higher tiers override lower tiers:
 
 1. **Entity annotations** (highest priority) — per-entity overrides set
    via catalog annotations with the prefix
-   `scorecard.io/<providerId>.thresholds.rules.<key>`. Merged on top of
-   the provider-level thresholds by `mergeEntityAndProviderThresholds()`.
-2. **`app-config.yaml` overrides** (middle priority) — cluster-wide
-   overrides configured at `scorecard.plugins.<providerId>.thresholds`
-   in `app-config.yaml`. Read by `getThresholdsFromConfig()` and stored
-   per provider ID in `ThresholdResolver.configuredThresholds`.
+   `scorecard.io/<metricId>.thresholds.rules.<key>` (full metric ID).
+   Merged on top of resolved thresholds by
+   `mergeEntityAndMetricThresholds()`.
+2. **`app-config.yaml` overrides** (middle priority) — administrator
+   overrides by configuration, most specific level wins
+   (datasource < provider < metric), complete replace of metric defaults.
+   - metric:
+     `plugins.<datasource>.metricProviders.<providerName>.metrics.<metricName>.thresholds`
+   - provider: `plugins.<datasource>.metricProviders.<providerName>.thresholds`
+   - datasource: `plugins.<datasource>.thresholds`
+     Config keys are local names (no datasource prefix).
 3. **Metric defaults** (lowest priority) — the `thresholds` field on
    each `Metric` object returned by `getMetrics()`. Used when no config
    or entity override exists.
 
-**Current limitation:** config overrides (tier 2) are keyed by
-`providerId`, not by individual `metricId`. This means all metrics
-within a multi-metric provider (e.g., SonarQube with 12 metrics, or
-Dependabot with 4 metrics) share a single `app-config.yaml` threshold
-override. Cluster administrators cannot customize thresholds for
-individual metrics within the same provider via config alone — they
-must use entity-level annotations (tier 1) for per-metric granularity.
-
-In `ThresholdResolver.setConfiguredThresholds()`, the config is
-validated against `metrics[0].type`, which is safe only because the
-generic constraint guarantees type homogeneity within a provider.
-However, this still applies one threshold config to all metrics in
-that provider.
+Provider IDs are either `datasource` (batch providers such as filecheck)
+or `datasource.<providerName>`. Metric IDs must always be
+`datasource.<metricName>`.
 
 ### Review guidance for threshold changes
 
 When reviewing changes to `ThresholdResolver`, the `Metric` type,
-`mergeEntityAndProviderThresholds`, or `getThresholdsFromConfig`:
+`mergeEntityAndMetricThresholds`, or `getThresholdsFromConfig`:
 
 - **Flag changes that affect threshold granularity as high severity.**
-  Any change to how thresholds are keyed (per-provider vs per-metric)
-  or how overrides are resolved directly impacts how cluster
-  administrators configure the scorecard in `app-config.yaml`. This is
-  not a theoretical concern — it determines whether operators can tune
-  thresholds for individual metrics.
+  Any change to how thresholds are keyed or how overrides are resolved
+  directly impacts how cluster administrators configure the scorecard
+  in `app-config.yaml`.
 - **Verify all three resolution tiers.** A change that fixes one tier
   (e.g., improving config overrides) can break another (e.g.,
   entity annotation merging) if the data structures change.
-- **Check config path strings.** The config path
-  `scorecard.plugins.${providerId}.thresholds` is constructed as a
-  string. Changes to provider IDs or config schema must update both the
-  path construction and `config.d.ts`.
+- **Check config path strings.** Threshold paths are built via
+  `getMetricThresholdsConfigPath()`. Changes to provider/metric IDs or
+  config schema must update both path helpers and `config.d.ts`.
 
 ### Key files
 
-| File | Package | Role |
-|---|---|---|
-| `ThresholdResolver.ts` | `scorecard-backend` | Resolves thresholds using the three-tier chain |
-| `Metric.ts` | `scorecard-common` | Defines `Metric`, `MetricType`, and `MetricValue` types |
-| `MetricProvider.ts` | `scorecard-node` | Defines the `MetricProvider<T>` interface |
-| `mergeEntityAndProviderThresholds.ts` | `scorecard-backend` | Merges entity annotation overrides with provider thresholds |
-| `getThresholdsFromConfig.ts` | `scorecard-node` | Reads and validates threshold config from `app-config.yaml` |
-| `DependabotConfig.ts` | `scorecard-backend-module-dependabot` | Dependabot provider metric and threshold definitions |
-| `SonarQubeConfig.ts` | `scorecard-backend-module-sonarqube` | SonarQube provider metric and threshold definitions |
-| `OpenSSFConfig.ts` | `scorecard-backend-module-openssf` | OpenSSF provider metric and threshold definitions |
-| `FilecheckConfig.ts` | `scorecard-backend-module-filecheck` | Filecheck provider metric and threshold definitions |
+| File                                | Package                               | Role                                                        |
+| ----------------------------------- | ------------------------------------- | ----------------------------------------------------------- |
+| `ThresholdResolver.ts`              | `scorecard-backend`                   | Resolves thresholds using the three-tier chain              |
+| `Metric.ts`                         | `scorecard-common`                    | Defines `Metric`, `MetricType`, and `MetricValue` types     |
+| `MetricProvider.ts`                 | `scorecard-node`                      | Defines the `MetricProvider<T>` interface                   |
+| `mergeEntityAndMetricThresholds.ts` | `scorecard-backend`                   | Merges entity annotation overrides with metric thresholds   |
+| `getThresholdsFromConfig.ts`        | `scorecard-node`                      | Reads and validates threshold config from `app-config.yaml` |
+| `DependabotConfig.ts`               | `scorecard-backend-module-dependabot` | Dependabot provider metric and threshold definitions        |
+| `SonarQubeConfig.ts`                | `scorecard-backend-module-sonarqube`  | SonarQube provider metric and threshold definitions         |
+| `OpenSSFConfig.ts`                  | `scorecard-backend-module-openssf`    | OpenSSF provider metric and threshold definitions           |
+| `FilecheckConfig.ts`                | `scorecard-backend-module-filecheck`  | Filecheck provider metric and threshold definitions         |

workspaces/scorecard/app-config.local.EXAMPLE.yaml

@@ -77,7 +77,7 @@ scorecard:
             - key: error
               expression: '<51'
               color: '#be1ec7' # purple
-  plugins:
+  metricProviders:
     jira:
       openIssues:
         schedule:
@@ -90,7 +90,6 @@ scorecard:
         #   customFilter: priority = High
     github:
       openPRs:
-        # Example threshold overrides; remove to use provider defaults.
         thresholds:
           rules:
             - key: success

workspaces/scorecard/app-config.production.yaml

@@ -61,7 +61,7 @@ catalog:
 #       description: Open PRs across owned entities, grouped by status.
 #       type: statusGrouped
 #       metricId: github.openPRs
-#   plugins:
+#   metricProviders:
 #     github:
 #       openPRs:
 #         schedule:

workspaces/scorecard/app-config.yaml

@@ -274,7 +274,7 @@ scorecard:
       type: statusGrouped
       description: This KPI is provide information about whether the license file exists in the repository.
       metricId: filecheck.license
-  plugins:
+  metricProviders:
     jira:
       openIssues:
         schedule:
@@ -288,10 +288,12 @@ scorecard:
           timeout: { minutes: 10 }
           initialDelay: { seconds: 10 }
     filecheck:
-      files:
-        license: 'LICENSE'
-        codeowners: 'CODEOWNERS'
-      schedule:
-        frequency: { minutes: 5 }
-        timeout: { minutes: 10 }
-        initialDelay: { seconds: 10 }
+      fileExistence:
+        options:
+          files:
+            license: 'LICENSE'
+            codeowners: 'CODEOWNERS'
+        schedule:
+          frequency: { minutes: 5 }
+          timeout: { minutes: 10 }
+          initialDelay: { seconds: 10 }

workspaces/scorecard/plugins/scorecard-backend-module-dependabot/README.md

@@ -17,15 +17,16 @@ All four severity metrics share the same default thresholds. Default thresholds
 scorecard:
   plugins:
     dependabot:
-      alertsCritical:
-        thresholds:
-          rules:
-            - key: success
-              expression: '<1'
-            - key: warning
-              expression: '1-7'
-            - key: error
-              expression: '>7'
+      metricProviders:
+        alertsCritical:
+          thresholds:
+            rules:
+              - key: success
+                expression: '<1'
+              - key: warning
+                expression: '1-7'
+              - key: error
+                expression: '>7'
 ```
 
 Replace `alertsCritical` with `alertsHigh`, `alertsMedium`, or `alertsLow` for the other severity metrics. See [threshold configuration](../scorecard-backend/docs/thresholds.md) for custom configuration.