feat: emit OTel log signals on unrouted requests — GH#4705

[balhar-jakub]

Closes #4705

Summary

Extend OtelRequestFilter to emit OpenTelemetry log signals when HTTP requests cannot be routed due to unknown Service ID or unavailable service instances. Fixes the response code timing bug.

Changes (3 areas, 3 commits)

Area A — OtelRequestContext attributes (gateway-service)

• 3 new fluent-setter methods: statusCode(int), errorType(String), errorMessage(String)
• 3 new constants: http.response.status_code, error.type, error.message
• 3 unit tests verifying string conversion and storage
• Existing responseCode() → service.response_code preserved for backward compatibility

Area B — OtelRequestFilter error handling (apiml)

• .doOnError() hooks for NoResourceFoundException → 404, error.type="Service not onboarded"
• .doOnError() hooks for ServiceNotAccessibleException → 503, error.type="Service instance not available"
• Timing fix: response code now set inside doFinally (on ON_COMPLETE) before issue(), fixing the bug where issue() ran before responseCode() was set
• 2 unit tests covering both error scenarios with StepVerifier

Area C — Documentation (otel/README.md)

• New "HTTP Error Attributes" section with attribute table (key, type, description)
• Two error scenarios documented: Unknown Service ID (404) and Service Instances Down (503)
• Example Prometheus/Alertmanager alert expressions for monitoring

Build Validation

• ./gradlew clean build — BUILD SUCCESSFUL (410 tasks: 270 executed, 92 from cache, 48 up-to-date)
• No new test failures

(Architectural review to follow as a PR comment)

[balhar-jakub]

Architectural Review — PR #4725 (Area A)

Verdict: APPROVED ✅

Summary

Area A of #4705 — adding three new OTel log signal attributes to OtelRequestContext. Clean, minimal change with no architectural concerns.

Review Checklist

Criterion	Status	Notes
Design compliance	✅	Matches the approved design document exactly
Structural integrity	✅	Follows existing fluent-setter pattern consistently
API contracts	✅	Additive only — no breaking changes
Test coverage	✅	3 new tests, one per method
Backward compatibility	✅	responseCode() → service.response_code preserved
Attribute naming	✅	Follows OTel semantic conventions

Detailed Review

1. Constants (lines 51-53)

• OTEL_ATTRIBUTE_STATUS_CODE = "http.response.status_code" — standard OTel HTTP semantic convention. Correct.
• OTEL_ATTRIBUTE_ERROR_TYPE = "error.type" — standard OTel error semantic convention. Correct.
• OTEL_ATTRIBUTE_ERROR_MESSAGE = "error.message" — follows the same naming pattern as existing auth.error.message. Consistent with the codebase. (Note: OTel spans use exception.message, but this is a log signal — error.message is more general and appropriate.)

2. Methods (lines 94-104)

• All three methods follow the exact same fluent-setter pattern as existing methods: put(key, value) → return this. This is consistent and correct.
• statusCode(int) uses String.valueOf(status) — matches responseCode() pattern.
• errorType(String) and errorMessage(String) pass through directly — no transformation needed for string-valued attributes.

3. Tests (lines 90-105 of test file)

• givenOtelContext_whenSetStatusCode_thenTransformToString: Tests statusCode(503) → "503". Covers the String.valueOf() conversion path.
• givenOtelContext_whenSetErrorType_thenStoreIt: Tests errorType("Service not onboarded") — validates one of the exact error strings from the acceptance criteria.
• givenOtelContext_whenSetErrorMessage_thenStoreIt: Tests errorMessage("Service instance not available") — validates the other error string.

All three follow the existing test pattern (OtelRequestContext.of(exchange).method() → getValue(key)).

No Issues Found

• No new dependencies introduced
• No changes to the filter chain or reactive pipeline
• No threading concerns — AttributesBuilder is already used in the same pattern throughout
• No license header issues

Downstream Notes

Area B (OtelRequestFilter error handling) will consume these methods via .doOnError() hooks. The fluent API is ready for that integration:

otelContext.statusCode(404)
    .errorType("Service not onboarded")
    .errorMessage("Service " + attemptedService + " is not registered");

[balhar-jakub]

Architectural Review — Area B (OtelRequestFilter, commit f6e9f44)

Status: APPROVED ✅

Summary

Area B extends OtelRequestFilter.filterInternal() with doOnError hooks for NoResourceFoundException (404) and ServiceNotAccessibleException (503), and fixes a timing bug where service.response_code was set after the OTel log signal was already issued.

---

1. Design Compliance ✅

The implementation follows the Step 2 design exactly:

• doOnError(NoResourceFoundException) → statusCode(404), errorType("Service not onboarded"), custom errorMessage
• doOnError(ServiceNotAccessibleException) → statusCode(503), errorType("Service instance not available"), errorMessage from exception
• Attributes populated before issue() in the reactive chain

2. Timing Fix Correctness ✅

Before (bug):

.doFinally(signalType -> otelContext.issue())          // log issued FIRST
.then(Mono.fromRunnable(() -> ... responseCode ...))   // responseCode set AFTER

After (fixed):

.doFinally(signalType -> {
    if (signalType == SignalType.ON_COMPLETE) {
        // set responseCode from actual response BEFORE issue()
        Optional.ofNullable(exchange.getResponse())
            .map(ServerHttpResponse::getStatusCode)
            .map(HttpStatusCode::value)
            .ifPresent(otelContext::responseCode);
    }
    otelContext.issue();   // log now includes responseCode
});

The SignalType.ON_COMPLETE gate is correct:

• ON_COMPLETE → response has a status code → populate it ✓
• ON_ERROR → response may be null or unset → skip ✓
• ON_CANCEL → subscription cancelled → skip ✓

3. Reactive Chain Structure ✅

• doOnError hooks fire BEFORE doFinally in Reactor order — attributes are set before issue() runs
• Exceptions propagate downstream (not swallowed) — confirmed by StepVerifier.expectError() in tests
• Generic RuntimeException (existing test) still triggers issue() via doFinally without error attributes — correct behavior

4. Error Message Construction ✅

• attemptedService extracted from pathElements.get(1).value() — same pattern used in setDefaults() (line 126), consistent with existing code
• Root path / fallback → "gateway" — correct (root serves Gateway homepage)
• ServiceNotAccessibleException.getMessage() — canonical source for the error message

5. Backward Compatibility ✅

• service.response_code still populated on success path (ON_COMPLETE)
• Existing test givenMultipleFiltersInPipeline_whenStarted_thenLogOnceResponse verifies service.response_code = "503" still works
• New http.response.status_code is additive, not a replacement

6. Test Coverage ✅

Test	Scenario	Coverage
givenUnknownService_whenNoResourceFound_thenSet404AndErrorType	NoResourceFoundException	statusCode(404), errorType, errorMessage, issue()
givenServiceNotAccessible_whenCalled_thenSet503AndErrorType	ServiceNotAccessibleException	statusCode(503), errorType, errorMessage, issue()
givenRequest_whenFailed_thenIssueLog (existing)	Generic RuntimeException	issue() still called, no attr overrides
givenMultipleFiltersInPipeline_whenStarted_thenLogOnceResponse (existing)	Success path	response_code still set, issue() once

7. Potential Risks (Low)

• Root path edge case: pathElements.size() > 1 gate → single-element path falls back to "gateway". Verified correct.
• doOnError supertype matching: NoResourceFoundException and ServiceNotAccessibleException are in unrelated hierarchies — no ordering conflicts. ✅

Verdict

Clean implementation. No structural concerns. Ready for CI validation.

— Architect

[balhar-jakub]

Architectural Review — PR #4725 for #4705

Reviewer: Architect (Hermes pipeline — Steps 5-6)
Build: ./gradlew clean build — BUILD SUCCESSFUL (410 tasks, 2m 24s)
Commits: 3 (4bca094, f6e9f44, 44848fc)
Files changed: 5 (+154 / −7)

Design Compliance ✅

All implementation decisions match the design specification:

Design Decision	Implementation	Status
Extend OtelRequestFilter (not new WebFilter)	.doOnError() hooks added to existing filterInternal()	✅
Distinguish 404 vs 503 via exception type	NoResourceFoundException → 404, ServiceNotAccessibleException → 503	✅
Fix response code timing bug	Moved responseCode() into doFinally(ON_COMPLETE) before issue()	✅
Detection in OtelRequestFilter	Error hooks are scoped inside filterInternal() — reactive, non-blocking	✅
user.id null for unknown services	Accepted — setDefaults() handles what's available; null attributes are omitted by OTel	✅

Structural Integrity ✅

• 5 files, 154 insertions — changes are surgically focused on the OtelRequestFilter → OtelRequestContext path
• No new dependencies, no module boundary violations
• OtelRequestContext setters follow existing fluent pattern (responseCode(), serviceId() etc.)
• doOnError hooks are scoped per-exception-type — no catch-all, no swallowed errors
• SignalType.ON_COMPLETE guard ensures responseCode() only fires on success — error paths set statusCode() directly

API Contracts ✅

OTel Attribute	Semantic Convention	Type	Usage
http.response.status_code	HTTP semantic convention	int	Reserved OTel key — correct semantic usage
error.type	Exception semantic convention	string	Machine-readable error category
error.message	Exception semantic convention	string	Human-readable description

• service.response_code (existing) is preserved — backward compatible
• All attributes set as String.valueOf() — consistent with existing pattern

Test Coverage ✅

• OtelRequestFilterTest: 2 new tests (givenUnknownService_whenNoResourceFound_thenSet404AndErrorType, givenServiceNotAccessible_whenCalled_thenSet503AndErrorType) — both use StepVerifier to verify reactive error propagation + attribute values
• OtelRequestContextTest: 3 new tests for statusCode(), errorType(), errorMessage() — verify string conversion
• All 7 OtelRequestFilter tests pass (5 existing + 2 new)

Documentation ✅

otel/README.md additions are clear, well-structured, and include:

• Attribute reference table (key, type, description)
• Both error scenarios with expected values
• Distinction between http.response.status_code (client-facing) and service.response_code (downstream)
• Practical Prometheus/Alertmanager alert expressions

Verdict: APPROVED ✅

No architectural concerns. Implementation is a clean, minimal extension of existing infrastructure. Ready for CI validation.

---

Hermes pipeline: architect Steps 5-6 — t_383a1bcb

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[balhar-jakub]

QA Review: PR #4725

Impact: gateway-service (CRITICAL) + apiml filter chain (CRITICAL)
Author: @balhar-jakub
Branch: hermes/gh4705 → v3.x.x
Architect review: APPROVED (t_383a1bcb)

---

Acceptance Criteria Verification (Issue #4705)

#	Criterion	Status	Evidence
1	Log is generated on unknown service / service down	✅ PASS	doOnError hooks in OtelRequestFilter + issue() called in doFinally
2	http.response.status_code records 404 or 503	✅ PASS	statusCode(404) for NoResourceFoundException, statusCode(503) for ServiceNotAccessibleException
3	service.id records attempted service name	✅ PASS	setDefaults() already sets service.id from path element (line 126/134 of OtelRequestFilter) — works for both known and unknown services
4	error.type = "Service not onboarded" / "Service instance not available"	✅ PASS	Exact strings match issue spec
5	Acceptance tests	✅ PASS	2 new unit tests in OtelRequestFilterTest (404 + 503 scenarios with StepVerifier) + 3 new tests in OtelRequestContextTest
6	Documentation updated	✅ PASS	otel/README.md — HTTP Error Attributes section with attribute table, error scenarios, Prometheus alert examples

All 6 acceptance criteria: VERIFIED ✅

---

Files Changed (5 files, +154/-7)

File	Component	Risk
OtelRequestContext.java	gateway-service	CRITICAL — 3 new fluent setters + constants
OtelRequestFilter.java	apiml filter chain	CRITICAL — error hooks + timing fix
OtelRequestContextTest.java	gateway-service tests	LOW — 3 new unit tests
OtelRequestFilterTest.java	apiml tests	LOW — 2 new unit tests
otel/README.md	documentation	LOW

Test Adequacy

✅ Adequate

• 3 unit tests for OtelRequestContext (statusCode, errorType, errorMessage)
• 2 unit tests for OtelRequestFilter (404 NoResourceFoundException + 503 ServiceNotAccessibleException)
• Tests verify both attribute values AND that issue() is called
• StepVerifier used correctly for reactive error scenarios

CI Status

44/44 checks PASS — all green (BuildAndTest, CITests, CITestsHA variants, SonarCloud, DCO, E2EUI, etc.)

Timing Fix Verification

The old code ran issue() before responseCode() — the response code was never captured for error paths. The new code moves response code capture into doFinally (before issue()), with ON_COMPLETE guard for success paths. Error paths set statusCode explicitly via doOnError hooks. This is correct.

Pavel's Lens (8 Rules)

Rule	Status	Notes
1. Config Consistency	N/A	No new config properties
2. Deduplication	✅	Constants and methods in correct locations
3. Null Safety	✅	Objects.toString(e.getMessage(), "No available instances") — proper fallback
4. Test Parametrization	✅	Only 2 tests, distinct scenarios — no parametrization needed
5. Security Boundaries	✅	No security-sensitive changes
6. z/OS Awareness	N/A	No startup/connection changes
7. Log Quality	✅	Error messages include service name for debugging
8. TODO Tracking	✅	No TODOs in diff

Minor Notes (non-blocking)

1. user.id in error logs: The issue lists user.id as a log signal attribute. In error paths (404/503), the request fails before authentication, so user.id is not available. This is expected — the existing userId() setter is called from the auth flow (ZaasSchemeTransformApi), which doesn't run for unroutable requests. No action needed.

2. attemptedService variable: Used only in the 404 error message. The 503 path uses the exception message directly (which already contains the service name). Consistent and clean.

---

Verdict: ✅ QA PASSED

All acceptance criteria verified. All CI checks green. No blocking issues. Pavel's Lens: all 8 rules checked, no issues.

Recommendation: Ready to merge.

[balhar-jakub]

QA Review — PR #4725 for #4705

Reviewer: QA (Hermes pipeline)
Date: 2026-06-27
Verdict: ✅ APPROVED

---

Acceptance Criteria Verification

Criterion	Status	Evidence
Log generated on unrouted request	✅	issue() called in doFinally for all signal types
http.response.status_code = 404 for unknown service	✅	statusCode(404) in doOnError(NoResourceFoundException)
http.response.status_code = 503 for unavailable service	✅	statusCode(503) in doOnError(ServiceNotAccessibleException)
error.type = "Service not onboarded"	✅	Exact string in doOnError(NoResourceFoundException)
error.type = "Service instance not available"	✅	Exact string in doOnError(ServiceNotAccessibleException)
service.id records attempted service name	✅	Extracted from pathElements.get(1).value()
Acceptance tests	✅	5 new tests (3 OtelRequestContext + 2 OtelRequestFilter)
Documentation updated	✅	otel/README.md with attribute table + monitoring examples

Test Coverage Analysis

New Tests (5):

Test	Module	Coverage
givenOtelContext_whenSetStatusCode_thenTransformToString	gateway-service	statusCode(503) → "503"
givenOtelContext_whenSetErrorType_thenStoreIt	gateway-service	errorType("Service not onboarded") storage
givenOtelContext_whenSetErrorMessage_thenStoreIt	gateway-service	errorMessage("Service instance not available") storage
givenUnknownService_whenNoResourceFound_thenSet404AndErrorType	apiml	Full 404 error path with StepVerifier
givenServiceNotAccessible_whenCalled_thenSet503AndErrorType	apiml	Full 503 error path with StepVerifier

Existing Tests Preserved:

• All 5 existing OtelRequestFilterTest tests pass
• All existing OtelRequestContextTest tests pass
• Full CI suite: 44 checks, all PASS

Code Quality

Positive:

• Follows existing fluent-setter pattern (responseCode(), serviceId(), etc.)
• Proper null handling: Objects.toString(e.getMessage(), "No available instances")
• Timing fix correctly moves responseCode() into doFinally(ON_COMPLETE) before issue()
• SignalType.ON_COMPLETE gate prevents setting responseCode on error paths
• Backward compatible: service.response_code still populated on success path

SonarCloud:

• Quality Gate: PASSED
• 0 new issues
• 100% coverage on new code
• 0% duplication on new code

Documentation Review

otel/README.md additions:

• ✅ Attribute reference table (key, type, description)
• ✅ Both error scenarios documented (404 and 503)
• ✅ Distinction between http.response.status_code and service.response_code
• ✅ Practical Prometheus/Alertmanager alert expressions

Risk Assessment

Low Risk:

• Additive changes only — no breaking changes to existing API
• Error hooks are scoped per-exception-type — no catch-all
• pathElements.size() > 1 guard handles edge cases correctly

Verdict

PR #4725 fully implements issue #4705 acceptance criteria. Implementation is clean, well-tested, and backward compatible. All CI checks pass. Ready for merge.

Recommendation: ✅ APPROVE and MERGE

---

QA review by Hermes pipeline — task t_a6343f82