Skip to content

fix: connect on Python 3.13+ and survive CA endpoint rate limits - #148

Merged
cayossarian merged 2 commits into
SpanPanel:mainfrom
brunocramos:fix/strict-x509-and-ca-retry
Aug 23, 2026
Merged

fix: connect on Python 3.13+ and survive CA endpoint rate limits#148
cayossarian merged 2 commits into
SpanPanel:mainfrom
brunocramos:fix/strict-x509-and-ca-retry

Conversation

@brunocramos

Copy link
Copy Markdown
Contributor

Summary

Two client-side defects prevent SpanMqttClient from connecting to healthy panels. Both surface as a generic SpanPanelConnectionError, so the panel looks unreachable even though every REST endpoint responds normally.

1. TLS handshake rejected under VERIFY_X509_STRICT

SPAN panels serve a minimal self-signed CA that omits the Authority Key Identifier X.509v3 extension. Python 3.13 enabled VERIFY_X509_STRICT by default, and that flag rejects the chain during verification:

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: Missing Authority Key Identifier

Confirmed against a panel on spanos3/r202621/05. The CA it serves carries only Basic Constraints and Key Usage — no AKI.

_build_ssl_context now clears VERIFY_X509_STRICT. The checks that matter here are unchanged: the only trust anchor is the panel's own CA, fetched over the local network immediately before use, and hostname, signature and expiry verification all stay enabled.

2. A single HTTP 429 aborts setup

The panel rate-limits GET /api/v2/certificate/ca (~7 req/s observed) and returns 429 once the limit is hit. A reconnect storm, or simply another client polling the same panel, is enough to trigger it. download_ca_cert issued one request and raised on any non-200, turning a transient condition into a hard setup failure needing a manual reload.

It now retries 429s with exponential backoff, honouring Retry-After when present and falling back to the backoff curve when the header is missing or malformed. Non-429 failures still fail fast. max_attempts and backoff_s are parameters, so callers can tune or disable it.

Worth noting: the panel sends no Retry-After on these 429s, so the fallback is the path that actually runs today.

Tests

tests/test_ssl_context.py (new) mints an AKI-less CA plus a leaf, runs a throwaway TLS server, and performs a real handshake:

  • succeeds with the fix
  • fails with SSLCertVerificationError: ... Authority Key Identifier when VERIFY_X509_STRICT is set

That second test is deliberately a canary — if a future Python stops rejecting these chains it fails, signalling the workaround can be dropped.

Retry coverage in tests/test_detection_auth.py: success-after-429, attempt exhaustion, Retry-After handling, malformed and negative header values, and fail-fast for non-429.

376 passed
ruff: All checks passed
mypy: Success: no issues found

Verification

Against a real panel (spanos3/r202621/05, model 00200): CA download, SSL context construction, MQTT CONNACK, and 132 live messages received in an 8-second window. Before the change the same panel failed at the TLS handshake.

Note on scope

This does not address GET /api/v1/circuits returning HTTP 500 on this firmware, or the absence of energy.ebus.device.circuit nodes over MQTT — both are server-side and can't be fixed from the client. Reported separately to SPAN.

…imits

Two client-side defects prevented the MQTT transport from establishing a
connection to healthy panels.

1. TLS handshake rejected under VERIFY_X509_STRICT

   SPAN panels serve a minimal self-signed CA that omits the Authority Key
   Identifier X.509v3 extension. Python 3.13 enabled VERIFY_X509_STRICT by
   default, and that flag rejects such a chain during verification with
   "Missing Authority Key Identifier". The failure surfaces as a generic
   SpanPanelConnectionError, so the integration reports the panel as
   unreachable even though every REST endpoint responds normally.

   _build_ssl_context now clears VERIFY_X509_STRICT. The checks that matter
   for this connection are unchanged: the only trust anchor is the panel's
   own CA, fetched over the local network immediately before use, and
   hostname, signature and expiry verification all remain enabled.

2. A single HTTP 429 aborted setup

   The panel rate-limits GET /api/v2/certificate/ca and returns 429 once the
   limit is hit. A reconnect storm, or simply another client polling the same
   panel, is enough to trigger it. download_ca_cert issued one request and
   raised on any non-200, turning a transient condition into a hard setup
   failure that required a manual reload to clear.

   download_ca_cert now retries 429 responses with exponential backoff,
   honouring Retry-After when present and falling back to the backoff curve
   when the header is missing or malformed. Non-429 failures still fail fast.
   Attempt count and base delay are parameters, so callers can tune or
   disable the behaviour.

Tests
  - tests/test_ssl_context.py performs a real TLS handshake against a local
    server presenting an AKI-less chain, asserting it succeeds with the fix
    and fails with SSLCertVerificationError when VERIFY_X509_STRICT is set.
    That second test is deliberately a canary: if a future Python stops
    rejecting these chains it fails, signalling the workaround can be
    dropped.
  - Retry coverage coverage for the success-after-429 path, exhaustion,
    Retry-After handling, malformed and negative header values, and the
    fail-fast path for non-429 errors.

Verified against a panel on spanos3/r202621/05 (model 00200), where MQTT
previously failed to connect and now completes the handshake and streams.

376 tests pass; ruff and mypy are clean on the changed files.
@cayossarian
cayossarian self-requested a review August 23, 2026 03:27
@cayossarian

Copy link
Copy Markdown
Member

Thank you for this, @brunocramos — and sorry for the slow turnaround. This is an unusually well-put-together contribution: a real panel reproduction, a documented firmware version, and tests that mint an AKI-less CA and run an actual TLS handshake rather than asserting on flags. The canary test that fails if a future Python stops rejecting these chains is a particularly nice touch, and exactly the kind of thing that keeps a workaround from outliving its cause.

A note on where this is landing. main has just moved to 3.0.0, which splits the wire-format parser out of the client into separately-installable schema adapters and raises the Python floor to 3.14. There is no point cutting a release off the 2.6.x line, so I have retargeted this at 3.0.0 rather than merging it where you opened it. I have verified the merge is clean and the full suite passes on the new tree.

That also explains the build-check failure you may have seen here — it is not your change. main at 2.6.4 pinned bare twine and locked 6.2.0, while hatchling is resolved fresh at build time and now emits Metadata-Version: 2.5, which 6.2.0 rejects. 3.0.0 already carries the twine>=7.0 fix, so this goes green on the new base.

Two things worth recording, neither of them blocking:

The rate-limit retry is the half that changes behaviour for us. download_ca_cert raised on any non-200, and a reconnect storm or a second client polling the same panel was enough to turn a transient 429 into a hard setup failure. Honouring Retry-After with a backoff fallback is the right shape, and worst-case latency works out to 22.5s of backoff across the five attempts, which is comfortably inside what Home Assistant setup tolerates.

The VERIFY_X509_STRICT clear appears to be inert on this code path, and I would like to understand your reproduction better rather than assume either of us is wrong. _build_ssl_context constructs ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) directly rather than going through ssl.create_default_context(), and the strict flag is only set by the latter. Measured on 3.14.6:

bare SSLContext(PROTOCOL_TLS_CLIENT)   verify_flags=32768    STRICT set? False
create_default_context()               verify_flags=557088   STRICT set? True

That constructor has been there since the file was first added, so I cannot find a released version where this path had the flag set. Both TLS call sites go through _build_ssl_context, and the one place we do build a default context only ever serves plain-HTTP bootstrap URLs.

Your diagnosis of the panel's CA is certainly right — it does omit the Authority Key Identifier, and your test proves such a chain is rejected under strict verification. I am just not yet able to trace how the flag became set in the failing run. If you were on a different client, an older revision, or something in your environment was constructing the context differently, that would be useful to know, because it might mean there is a second path that still needs fixing. Either way I am keeping the change: it is correctly scoped, it costs nothing, and it documents a real constraint on the panel's certificate. The trust anchor is still only the panel's own CA, with hostname and signature checking untouched.

One follow-up I will handle on our side rather than ask you to: cryptography is not in our dev group. It only reaches the environment transitively through twine -> keyring -> secretstorage, which is Linux-only, so test_ssl_context.py runs in CI but silently skips for every macOS developer. With it added explicitly all 8 tests pass locally. Given this repository's own "a skip here is not a pass" stance, that gap is ours to close.

Thanks again — this will ship in 3.0.0 with your commits intact.

@cayossarian
cayossarian merged commit 189f5a2 into SpanPanel:main Aug 23, 2026
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants