fix: connect on Python 3.13+ and survive CA endpoint rate limits - #148
Conversation
…imits
Two client-side defects prevented the MQTT transport from establishing a
connection to healthy panels.
1. TLS handshake rejected under VERIFY_X509_STRICT
SPAN panels serve a minimal self-signed CA that omits the Authority Key
Identifier X.509v3 extension. Python 3.13 enabled VERIFY_X509_STRICT by
default, and that flag rejects such a chain during verification with
"Missing Authority Key Identifier". The failure surfaces as a generic
SpanPanelConnectionError, so the integration reports the panel as
unreachable even though every REST endpoint responds normally.
_build_ssl_context now clears VERIFY_X509_STRICT. The checks that matter
for this connection are unchanged: the only trust anchor is the panel's
own CA, fetched over the local network immediately before use, and
hostname, signature and expiry verification all remain enabled.
2. A single HTTP 429 aborted setup
The panel rate-limits GET /api/v2/certificate/ca and returns 429 once the
limit is hit. A reconnect storm, or simply another client polling the same
panel, is enough to trigger it. download_ca_cert issued one request and
raised on any non-200, turning a transient condition into a hard setup
failure that required a manual reload to clear.
download_ca_cert now retries 429 responses with exponential backoff,
honouring Retry-After when present and falling back to the backoff curve
when the header is missing or malformed. Non-429 failures still fail fast.
Attempt count and base delay are parameters, so callers can tune or
disable the behaviour.
Tests
- tests/test_ssl_context.py performs a real TLS handshake against a local
server presenting an AKI-less chain, asserting it succeeds with the fix
and fails with SSLCertVerificationError when VERIFY_X509_STRICT is set.
That second test is deliberately a canary: if a future Python stops
rejecting these chains it fails, signalling the workaround can be
dropped.
- Retry coverage coverage for the success-after-429 path, exhaustion,
Retry-After handling, malformed and negative header values, and the
fail-fast path for non-429 errors.
Verified against a panel on spanos3/r202621/05 (model 00200), where MQTT
previously failed to connect and now completes the handshake and streams.
376 tests pass; ruff and mypy are clean on the changed files.
|
Thank you for this, @brunocramos — and sorry for the slow turnaround. This is an unusually well-put-together contribution: a real panel reproduction, a documented firmware version, and tests that mint an AKI-less CA and run an actual TLS handshake rather than asserting on flags. The canary test that fails if a future Python stops rejecting these chains is a particularly nice touch, and exactly the kind of thing that keeps a workaround from outliving its cause. A note on where this is landing. That also explains the Two things worth recording, neither of them blocking: The rate-limit retry is the half that changes behaviour for us. The That constructor has been there since the file was first added, so I cannot find a released version where this path had the flag set. Both TLS call sites go through Your diagnosis of the panel's CA is certainly right — it does omit the Authority Key Identifier, and your test proves such a chain is rejected under strict verification. I am just not yet able to trace how the flag became set in the failing run. If you were on a different client, an older revision, or something in your environment was constructing the context differently, that would be useful to know, because it might mean there is a second path that still needs fixing. Either way I am keeping the change: it is correctly scoped, it costs nothing, and it documents a real constraint on the panel's certificate. The trust anchor is still only the panel's own CA, with hostname and signature checking untouched. One follow-up I will handle on our side rather than ask you to: Thanks again — this will ship in 3.0.0 with your commits intact. |
Summary
Two client-side defects prevent
SpanMqttClientfrom connecting to healthy panels. Both surface as a genericSpanPanelConnectionError, so the panel looks unreachable even though every REST endpoint responds normally.1. TLS handshake rejected under
VERIFY_X509_STRICTSPAN panels serve a minimal self-signed CA that omits the Authority Key Identifier X.509v3 extension. Python 3.13 enabled
VERIFY_X509_STRICTby default, and that flag rejects the chain during verification:Confirmed against a panel on
spanos3/r202621/05. The CA it serves carries onlyBasic ConstraintsandKey Usage— no AKI._build_ssl_contextnow clearsVERIFY_X509_STRICT. The checks that matter here are unchanged: the only trust anchor is the panel's own CA, fetched over the local network immediately before use, and hostname, signature and expiry verification all stay enabled.2. A single HTTP 429 aborts setup
The panel rate-limits
GET /api/v2/certificate/ca(~7 req/s observed) and returns 429 once the limit is hit. A reconnect storm, or simply another client polling the same panel, is enough to trigger it.download_ca_certissued one request and raised on any non-200, turning a transient condition into a hard setup failure needing a manual reload.It now retries 429s with exponential backoff, honouring
Retry-Afterwhen present and falling back to the backoff curve when the header is missing or malformed. Non-429 failures still fail fast.max_attemptsandbackoff_sare parameters, so callers can tune or disable it.Worth noting: the panel sends no
Retry-Afteron these 429s, so the fallback is the path that actually runs today.Tests
tests/test_ssl_context.py(new) mints an AKI-less CA plus a leaf, runs a throwaway TLS server, and performs a real handshake:SSLCertVerificationError: ... Authority Key IdentifierwhenVERIFY_X509_STRICTis setThat second test is deliberately a canary — if a future Python stops rejecting these chains it fails, signalling the workaround can be dropped.
Retry coverage in
tests/test_detection_auth.py: success-after-429, attempt exhaustion,Retry-Afterhandling, malformed and negative header values, and fail-fast for non-429.Verification
Against a real panel (
spanos3/r202621/05, model 00200): CA download, SSL context construction, MQTT CONNACK, and 132 live messages received in an 8-second window. Before the change the same panel failed at the TLS handshake.Note on scope
This does not address
GET /api/v1/circuitsreturning HTTP 500 on this firmware, or the absence ofenergy.ebus.device.circuitnodes over MQTT — both are server-side and can't be fixed from the client. Reported separately to SPAN.