fix: escalate peer read timeouts after max retries — GH#4777

[balhar-jakub]

Closes #4777

Problem

The peer-node-read-timeout-ms was too short (15s), causing infinite retry loops under load on z/OS. Read timeouts were never escalated to permanent errors.

Fix

• Added networkIssueCounter.hasReachedMax() gate to read timeout handling in ApimlPeerEurekaNode
• After maxPeerRetries consecutive read timeout failures → PermanentError (matching connection-failure behavior)
• Increased default peer-node-read-timeout-ms from 15000 → 30000ms
• Reset counter on successful replication
• 4 new parametrized tests covering escalation, reset, and unchanged behavior

Files changed

• apiml-common/.../ApimlPeerEurekaNode.java: hasReachedMax() gate in both process(task) and process(tasks)
• discovery-service/.../application.yml: timeout 15000 → 30000
• apiml/.../application.yml: timeout 15000 → 30000
• apiml-common/.../ReplicationTaskProcessorTest.java: 4 new tests (19 total)

Build: PASS (19/19 tests)

[balhar-jakub]

QA + Security Review (Pavel's Lens) — APPROVED ✓

Pavel's Lens Results

Rule	Check	Verdict
Rule 1: Config Consistency	Both application.yml files updated identically (15000→30000)	✓ PASS
Rule 2: Deduplication	hasReachedMax() gate in both process(task) and process(tasks) — minimal, necessary duplication	✓ PASS
Rule 3: Null Safety	networkIssueCounter is initialized field; getCountText() returns StringBuilder.toString() (never null)	✓ PASS
Rule 4: Test Parametrization	4 standalone tests covering escalation, reset, regression	✓ PASS
Rule 5: Security Boundaries	No auth/authz/TLS changes; error messages don't expose sensitive data	✓ PASS
Rule 6: z/OS Awareness	Timeout increase (30s) specifically for z/OS batch replication latency	✓ PASS
Rule 7: Log Quality	Clear log message with retry count; existing warning preserved	✓ PASS
Rule 8: TODO Tracking	No new TODOs introduced	✓ PASS

Acceptance Criteria Verification

• AC1 ✓: hasReachedMax() gate escalates to PermanentError after N retries in both methods — covered by tests
• AC2 ✓: Successful replication resets counter — covered by whenSuccessfulReplicationResetsReadTimeoutCounter test
• AC3 ✓: Connection failure behavior unchanged — covered by whenConnectionFailureRepeatedMultipleTimes_thenBehaviorUnchanged test
• AC4 ✓: 4 tests covering escalation, reset, and unchanged behavior (19/19 pass)
• AC5 ✓: Default timeout increased to 30000ms in both config files

Security Review

• No sensitive data in error messages (only generic "read timed out" and retry count)
• No new endpoints or auth changes
• No secrets/credentials exposed
• PermanentError drops task; peer catches up via registry sync — no DoS vector

Verdict: APPROVED — Ready for merge. All acceptance criteria met, all 8 Pavel rules passed, no security concerns.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[achmelo]

these 2 log messages are too similar and it would make it harder to debug. I would rephrase one of them and maybe also print the exception message, at least in the debug level

[balhar-jakub]

If we use it ourselves, wouldn't it make sense to assign it the standard log id according to standard? Even for the original one.

I will change the text so it is further away as well.