fix(azure): gate live tests, add offline coverage, fix api_config and retry errors - #696
Merged
vishal-bala merged 2 commits intoAug 19, 2026
Conversation
… retry errors The Azure OpenAI resource CI used has been retired, so the six live AzureOpenAITextVectorizer tests failed on every run. Because the redis-py matrix and notebook jobs both declare `needs: service-tests`, that one dead provider blocked all 31 downstream jobs. Gate the live params on AZURE_OPENAI_ENDPOINT / AZURE_OPENAI_API_KEY / OPENAI_API_VERSION all being set, so they skip cleanly today and re-enable themselves when the secrets are restored. Add the first offline coverage for this vectorizer, and NBVAL_SKIP the notebook's Azure cells, whose getpass fallback has no stdin under nbval. Two fixes found while writing the tests: - `_initialize_clients` no longer mutates the caller's api_config, and a partial api_config now falls back to the environment per key as the class docstring promises, instead of raising a bare KeyError. - `reraise=True` on the four retry decorators, so retry exhaustion surfaces the underlying ValueError instead of tenacity.RetryError. The docstrings already promised ValueError. Matches googlegenai.py and ollama.py.
Remove the CLAUDE.md bullet and the CONTRIBUTING.md note about the Azure env gate -- how the shared CI resource is managed is a maintainer decision, not contributor guidance. Drop the corresponding CONTRIBUTING.md pointer from the skip reason so it does not dangle. Drop the NBVAL_SKIP reason comment from the notebook's Azure config cell; CI mechanics do not belong in a teaching notebook. Collapse the six banner-box section headers in the new unit test file to plain one-line comments.
vishal-bala
marked this pull request as ready for review
August 19, 2026 14:51
vishal-bala
deleted the
tests/gate-azure-vectorizer-tests-add-mocked-unit-tests
branch
August 19, 2026 15:02
|
🚀 PR was released in |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The Azure OpenAI resource CI used is no longer available, with no replacement yet. The six live
AzureOpenAITextVectorizertests failed on every run, and because the redis-py matrix and notebook jobs both declareneeds: service-tests, one unavailable provider blocked all 31 downstream jobs — including a matrix that runsmake testwith no API keys and has zero Azure exposure.AzureOpenAITextVectorizeralso had no offline tests at all, unlike itsollamaandgoogle_genaisiblings. Gating the live tests without adding any would have taken its coverage to zero.What changed
tests/integration/test_vectorizers.pyskipifgate on the Azure entry in_vectorizer_paramsand_dtype_params._non_supported_dtype_paramsis deliberately not gated —dtypeis validated insuper().__init__before any client is built, so it passes with no network.tests/unit/test_azure_openai_vectorizer.pyopenaiis absent. Fakeopenaimodule viatypes.ModuleType, following the two existing sibling files.docs/user_guide/04_vectorizers.ipynb# NBVAL_SKIPon the three Azure cells. Required: without the credentials,os.environ.get(...) or getpass.getpass(...)falls through togetpass, which has no stdin under nbval. The pre-existing_azure_configuredguard cannot help — it is computed after the prompt.redisvl/utils/vectorize/text/azureopenai.py.github/workflows/test.ymlneeds: service-testsis untouched.Self-healing: the gate keys on
AZURE_OPENAI_ENDPOINT,AZURE_OPENAI_API_KEYandOPENAI_API_VERSIONall being truthy. An unset secret expands to"", so the tests skip today and re-enable with no code change once the credentials are available again. Thesecrets.AZURE_OPENAI_*lines stay wired up for exactly that reason. The notebook markers are unconditional and do have to be removed by hand.Behavior changes
Both are in
azureopenai.pyand both were found while writing the tests. Worth a look for release notes:reraise=Trueon the four@retrydecorators. Retry exhaustion now surfaces the underlyingValueErrorinstead oftenacity.RetryError. All four docstrings already promisedRaises: ValueError, so this fixes a contract violation. Matchesgooglegenai.pyandollama.py; nothing inredisvl/catchesRetryError._initialize_clientsno longer mutates the caller'sapi_config, and a partialapi_confignow falls back to the environment per key as the class docstring promises, instead of raising a bareKeyError. Uses thedict(api_config or {})pattern already ingooglegenai.py.Coverage: honestly
Lost — six live IDs stop running in CI:
test_default_dtype[Azure],test_vectorizer_dtype_assignment[Azure], and the fourvectorizer-fixture tests. Realistically that goes to zero with or without this PR, since the resource is unavailable; the gate removes a false signal rather than coverage. What the mocks cannot catch isopenaiSDK drift, deployment-vs-model semantics, real authentication, and true dimensionality.Mitigations — the
embeddings.create→response.data[i].embeddingshape still has live coverage throughOpenAITextVectorizer, which shares it and is unaffected. Andtest_fake_matches_real_openai_sdk_contractasserts the fake's shape against the installed SDK, so a reshape breaks this file too.Gained — several things the live tests never covered: the batching loop (they used 2 texts at default
batch_size=10, so it never ran), sync/async client symmetry, the three sequential credential guards,dtypebuffer widths, and that__init__fires one billable embed request.Verification
retry.sleeppatch would cost 30s+, and a guard test pins that exactly four methods are retry-decorated.black,isort,mypy,codespellclean. Notebook diff touches onlysourcearrays.Notes for review
openaibuilds them fromresponse.status_codeand the response body, and the Azure key travels only as theapi-keyheader. It would also have made Azure the only one of ~21 sites not interpolating{e}.azureopenai.py, three lines above the firstreraise=True. Adjacent, not overlapping. A comment here refers totests/unit/test_vectorizer_dim_errors.pyas added by that PR.Note
Medium Risk
Behavior changes in AzureOpenAITextVectorizer (api_config copying, per-key env fallback, retry reraise) affect callers relying on mutation or RetryError; CI gating is low risk but live Azure coverage drops until credentials return.
Overview
Unblocks CI after the Azure OpenAI deployment used in tests went away by skipping live Azure integration tests when
AZURE_OPENAI_ENDPOINT,AZURE_OPENAI_API_KEY, andOPENAI_API_VERSIONare unset, while adding a large offline unit suite with a fakeopenaimodule. Notebook validation no longer hangs ongetpassbecause the three Azure cells in04_vectorizers.ipynbare marked# NBVAL_SKIP.AzureOpenAITextVectorizernow copiesapi_configbefore popping credentials (no caller mutation), resolves each credential from config or env per key, and setsreraise=Trueon embed retries so failures surface asValueErrorinstead oftenacity.RetryError. Workflow comments document how live tests and notebooks re-enable when secrets return.Reviewed by Cursor Bugbot for commit 030769c. Bugbot is set up for automated code reviews on this repo. Configure here.