Skip to content

Say which stage of reaching the LLM endpoint failed - #186

Merged
Sev7eNup merged 1 commit into
mainfrom
fix/llm-connect-diagnostics
Aug 11, 2026
Merged

Say which stage of reaching the LLM endpoint failed#186
Sev7eNup merged 1 commit into
mainfrom
fix/llm-connect-diagnostics

Conversation

@Sev7eNup

Copy link
Copy Markdown
Owner

Why

An LLM endpoint that could not be reached reported "LLM endpoint did not respond within {TimeoutSeconds}s" — the same sentence a slow model produces. A single budget covered DNS, TCP, TLS and the model's answer, so on a profile with TimeoutSeconds: 360 an operator waited six minutes and was then pointed at the model, while the actual cause was a firewall or an untrusted certificate.

This came out of a real case: https://litellm-test.<internal>/v1, same network as NodePilot, no proxy involved. Both the settings probe and the chat failed, and the message said nothing that narrowed it down. Four distinct causes, one sentence, none of them named.

This does not fix that installation's connectivity — it makes the cause visible in seconds instead of minutes, and names it.

What

Reaching an endpoint is not the same kind of work as waiting for a model to think, so it no longer shares the budget.

Stage Deadline Message starts with
DNS 15 s (shared with TCP) LLM endpoint DNS:
TCP 15 s LLM endpoint TCP: — distinguishes dropped (firewall) from refused (nothing listening), and lists the addresses actually tried
TLS handshake 30 s LLM endpoint TLS:
Certificate LLM endpoint TLS: … certificate — points at the machine trust store
Model answer TimeoutSeconds accepted the request but sent no answer

LlmConnectGuard owns DNS and TCP and fails with the stage named. SocketsHttpHandler.ConnectTimeout bounds the rest — the only place that can bound the TLS handshake, because the callback hands back the raw transport stream and the handler negotiates on top of it.

The ordering is load-bearing and a test pins it. Because DNS and TCP always expire on their own, shorter deadline, a ConnectTimeout that fires can only mean the handshake stalled — that is how DescribeUnreachable names the TLS stage rather than guessing. If those two constants ever cross, the inference silently becomes wrong, so HandshakeTimeout_ExceedsTheConnectPhaseBudget fails the build instead.

Certificate rejections are called out separately and say where the CA belongs: a certificate a browser accepts on a workstation is not thereby trusted by the service account.

Deliberately no new configuration key. 15 s is far beyond any healthy lookup or handshake — Windows abandons an unanswered SYN after ~21 s on its own — so a value that would need raising means the network is broken, which is now exactly what the message says.

Two smaller things in the same pass:

  • The settings probe kept only HttpRequestException's generic "An error occurred while sending the request" and dropped the inner exception, which is where the stage or the certificate error lives. Its client timeout moves 30 s → 40 s so the handler's named stage wins the race instead of a bare "HttpClient.Timeout elapsed".
  • Resolved addresses and per-stage timings now go to Debug under the named category NodePilot.Ai.LlmConnect, switchable on its own via Serilog:MinimumLevel:Override. That is the line that settles "works from my machine, not from the service" — a stale AAAA record and a different DNS suffix look identical from the outside.

Verification

  • Full backend suite green: 5413 tests. 9 new (4 stage/logging tests through the real ConnectCallback, 5 on the classification).
  • dotnet build clean, no new warnings.
  • No frontend surface: this adds no setting, so no DTO, UI or i18n change.
  • One caveat, stated rather than glossed: a single API test failed once during a parallel solution run and passed both on isolated re-run and on a repeated full run. It did not reproduce, and I could not capture its name — same shape as the pre-existing flakiness the nightly harness retries for, not something this branch touches.

An endpoint that could not be reached reported "LLM endpoint did not respond
within {TimeoutSeconds}s" — the same sentence a slow model produces. One budget
covered DNS, TCP, TLS and the model's answer alike, so with a profile at 360 s
an operator waited six minutes and was then pointed at the model while the real
cause was a firewall or an untrusted certificate. Four causes, one message,
none of them named.

Reaching an endpoint is not the same kind of work as waiting for a model to
think, so it no longer shares the budget. LlmConnectGuard gives name resolution
and the TCP connect 15 s together and fails with the stage in the message;
SocketsHttpHandler.ConnectTimeout bounds the rest at 30 s, which is the only
place that can bound the TLS handshake — the callback hands back the raw
transport stream and the handler negotiates on top of it.

The two are ordered on purpose and a test pins it: because DNS and TCP always
expire on their own shorter deadline, a ConnectTimeout that fires can only mean
the handshake stalled, and DescribeUnreachable says so instead of guessing.
Certificate rejections are named separately and point at the machine trust
store, since a CA that a browser accepts on a workstation is not thereby
trusted by the service account.

The answer-timeout message now states what it knows: the request was on the
wire and the model went quiet — the one case where raising TimeoutSeconds is
the right answer.

Also: the settings probe kept only HttpRequestException's own generic message
and dropped the inner exception, which is where the stage or the certificate
error actually is. Its client timeout moves 30 s -> 40 s so the handler's
named stage wins the race instead of a bare "HttpClient.Timeout elapsed".

Resolved addresses and per-stage timings go to Debug under a named category,
NodePilot.Ai.LlmConnect, so the "works from my machine, not from the service"
case can be settled by log rather than by guesswork.
@Sev7eNup
Sev7eNup merged commit baf8ceb into main Aug 11, 2026
8 checks passed
@Sev7eNup
Sev7eNup deleted the fix/llm-connect-diagnostics branch August 11, 2026 20:45
Sev7eNup added a commit that referenced this pull request Aug 11, 2026
Carries the LLM connect diagnostics (#186): reaching an endpoint no longer
shares its budget with the model's answer, and a failure names the stage it
failed at — DNS, TCP, the TLS handshake, or the certificate — instead of
reporting every one of them as "did not respond within {TimeoutSeconds}s".

Bumped through `npm version --no-git-tag-version` per package rather than a
text replace over the tree, so only the top-level version and its lockfile root
entry move.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant