optimise prometheus scraping workflow

[DaanHoogland]

Description

This PR...

Fixes: #13586

1. Removes alertManager.recalculateCapacity() from PrometheusExporterImpl.updateMetrics(). This is the actual fix — it stops the exporter from forcing a full, thread-pool-churning, all-zone capacity recalculation on every single scrape. The exporter should just read CapacityDao, which AlertManagerImpl's own timer already keeps current. This alone should flatten the scrape_duration curve.
2. Reuses a single long-lived ExecutorService in AlertManagerImpl instead of creating/tearing one down on every recalculateHostCapacities()/recalculateStorageCapacities() call. Cheap, low-risk, and removes the thread-churn contributor even for the timer-driven path.

this should solve the issue, but claude suggested the following improvements as well:

3. Give the exporter's HttpServer an explicit bounded executor (httpServer.setExecutor(Executors.newFixedThreadPool(2))) so one slow scrape can't serialize/queue all others.
4. Add a short TTL/in-flight guard around updateMetrics() (e.g., skip recompute if last run was < N seconds ago, or synchronize so concurrent scrapes share one in-progress computation) so scrape frequency can never multiply backend load.
5. Instrument: log/measure updateMetrics() wall-clock time so the reporter (and CI) can confirm which sub-metric collector is actually slow and verify the fix closes the growth.

if anyone wants to, these are nice to haves/good first issue ;)

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 16.26%. Comparing base (549daae) to head (98f0570).

Additional details and impacted files

@@             Coverage Diff              @@
##               4.20   #13650      +/-   ##
============================================
- Coverage     16.26%   16.26%   -0.01%     
- Complexity    13434    13439       +5     
============================================
  Files          5667     5667              
  Lines        500731   500732       +1     
  Branches      60803    60804       +1     
============================================
- Hits          81455    81446       -9     
- Misses       410172   410183      +11     
+ Partials       9104     9103       -1     

Flag	Coverage Δ
uitests	4.14% <ø> (ø)
unittests	17.12% <100.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18763

[sonarqubecloud]

Quality Gate passed

Issues
39 New issues
0 Accepted issues

Measures
0 Security Hotspots
75.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud