Skip to content

fix(azure): gate live tests, add offline coverage, fix api_config and retry errors - #696

Merged
vishal-bala merged 2 commits into
mainfrom
tests/gate-azure-vectorizer-tests-add-mocked-unit-tests
Aug 19, 2026
Merged

fix(azure): gate live tests, add offline coverage, fix api_config and retry errors#696
vishal-bala merged 2 commits into
mainfrom
tests/gate-azure-vectorizer-tests-add-mocked-unit-tests

Conversation

@vishal-bala

@vishal-bala vishal-bala commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Why

The Azure OpenAI resource CI used is no longer available, with no replacement yet. The six live AzureOpenAITextVectorizer tests failed on every run, and because the redis-py matrix and notebook jobs both declare needs: service-tests, one unavailable provider blocked all 31 downstream jobs — including a matrix that runs make test with no API keys and has zero Azure exposure.

AzureOpenAITextVectorizer also had no offline tests at all, unlike its ollama and google_genai siblings. Gating the live tests without adding any would have taken its coverage to zero.

What changed

tests/integration/test_vectorizers.py skipif gate on the Azure entry in _vectorizer_params and _dtype_params. _non_supported_dtype_params is deliberately not gated — dtype is validated in super().__init__ before any client is built, so it passes with no network.
tests/unit/test_azure_openai_vectorizer.py New. 36 tests + 1 SDK-contract test that skips when openai is absent. Fake openai module via types.ModuleType, following the two existing sibling files.
docs/user_guide/04_vectorizers.ipynb # NBVAL_SKIP on the three Azure cells. Required: without the credentials, os.environ.get(...) or getpass.getpass(...) falls through to getpass, which has no stdin under nbval. The pre-existing _azure_configured guard cannot help — it is computed after the prompt.
redisvl/utils/vectorize/text/azureopenai.py Two fixes, below.
.github/workflows/test.yml Comments only. No structural change; needs: service-tests is untouched.

Self-healing: the gate keys on AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_API_KEY and OPENAI_API_VERSION all being truthy. An unset secret expands to "", so the tests skip today and re-enable with no code change once the credentials are available again. The secrets.AZURE_OPENAI_* lines stay wired up for exactly that reason. The notebook markers are unconditional and do have to be removed by hand.

Behavior changes

Both are in azureopenai.py and both were found while writing the tests. Worth a look for release notes:

  1. reraise=True on the four @retry decorators. Retry exhaustion now surfaces the underlying ValueError instead of tenacity.RetryError. All four docstrings already promised Raises: ValueError, so this fixes a contract violation. Matches googlegenai.py and ollama.py; nothing in redisvl/ catches RetryError.
  2. _initialize_clients no longer mutates the caller's api_config, and a partial api_config now falls back to the environment per key as the class docstring promises, instead of raising a bare KeyError. Uses the dict(api_config or {}) pattern already in googlegenai.py.

Coverage: honestly

Lost — six live IDs stop running in CI: test_default_dtype[Azure], test_vectorizer_dtype_assignment[Azure], and the four vectorizer-fixture tests. Realistically that goes to zero with or without this PR, since the resource is unavailable; the gate removes a false signal rather than coverage. What the mocks cannot catch is openai SDK drift, deployment-vs-model semantics, real authentication, and true dimensionality.

Mitigations — the embeddings.createresponse.data[i].embedding shape still has live coverage through OpenAITextVectorizer, which shares it and is unaffected. And test_fake_matches_real_openai_sdk_contract asserts the fake's shape against the installed SDK, so a reshape breaks this file too.

Gained — several things the live tests never covered: the batching loop (they used 2 texts at default batch_size=10, so it never ran), sync/async client symmetry, the three sequential credential guards, dtype buffer widths, and that __init__ fires one billable embed request.

Verification

  • Gate fires with the env vars unset and empty-string (the actual CI state) → 6 skipped; un-fires when set → tests reach the network, which is correct, since the gate is credential-presence not reachability.
  • New file: 36 passed + 1 skipped in ~2.7s. Speed is the assertion — a missed retry.sleep patch would cost 30s+, and a guard test pins that exactly four methods are retry-decorated.
  • black, isort, mypy, codespell clean. Notebook diff touches only source arrays.

Notes for review

  • A proposed message-redaction fix was reverted after verifying against the installed SDK that the API key is not reachable in provider exception messages — openai builds them from response.status_code and the response body, and the Azure key travels only as the api-key header. It would also have made Azure the only one of ~21 sites not interpolating {e}.
  • Merge-order note: fix: raise provider-specific errors from vectorizer _set_model_dims() #680 also touches azureopenai.py, three lines above the first reraise=True. Adjacent, not overlapping. A comment here refers to tests/unit/test_vectorizer_dim_errors.py as added by that PR.
  • feat(voyageai): add contextualized embeddings and refresh model support #692 touches the same two test/notebook files but appends well clear of these edits.

Note

Medium Risk
Behavior changes in AzureOpenAITextVectorizer (api_config copying, per-key env fallback, retry reraise) affect callers relying on mutation or RetryError; CI gating is low risk but live Azure coverage drops until credentials return.

Overview
Unblocks CI after the Azure OpenAI deployment used in tests went away by skipping live Azure integration tests when AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_API_KEY, and OPENAI_API_VERSION are unset, while adding a large offline unit suite with a fake openai module. Notebook validation no longer hangs on getpass because the three Azure cells in 04_vectorizers.ipynb are marked # NBVAL_SKIP.

AzureOpenAITextVectorizer now copies api_config before popping credentials (no caller mutation), resolves each credential from config or env per key, and sets reraise=True on embed retries so failures surface as ValueError instead of tenacity.RetryError. Workflow comments document how live tests and notebooks re-enable when secrets return.

Reviewed by Cursor Bugbot for commit 030769c. Bugbot is set up for automated code reviews on this repo. Configure here.

… retry errors

The Azure OpenAI resource CI used has been retired, so the six live
AzureOpenAITextVectorizer tests failed on every run. Because the redis-py
matrix and notebook jobs both declare `needs: service-tests`, that one dead
provider blocked all 31 downstream jobs.

Gate the live params on AZURE_OPENAI_ENDPOINT / AZURE_OPENAI_API_KEY /
OPENAI_API_VERSION all being set, so they skip cleanly today and re-enable
themselves when the secrets are restored. Add the first offline coverage for
this vectorizer, and NBVAL_SKIP the notebook's Azure cells, whose getpass
fallback has no stdin under nbval.

Two fixes found while writing the tests:

- `_initialize_clients` no longer mutates the caller's api_config, and a
  partial api_config now falls back to the environment per key as the class
  docstring promises, instead of raising a bare KeyError.
- `reraise=True` on the four retry decorators, so retry exhaustion surfaces
  the underlying ValueError instead of tenacity.RetryError. The docstrings
  already promised ValueError. Matches googlegenai.py and ollama.py.
Remove the CLAUDE.md bullet and the CONTRIBUTING.md note about the Azure
env gate -- how the shared CI resource is managed is a maintainer decision,
not contributor guidance. Drop the corresponding CONTRIBUTING.md pointer
from the skip reason so it does not dangle.

Drop the NBVAL_SKIP reason comment from the notebook's Azure config cell;
CI mechanics do not belong in a teaching notebook.

Collapse the six banner-box section headers in the new unit test file to
plain one-line comments.
@vishal-bala
vishal-bala marked this pull request as ready for review August 19, 2026 14:51
@vishal-bala vishal-bala added the auto:patch Increment the patch version when merged label Aug 19, 2026
@vishal-bala
vishal-bala merged commit 9aff344 into main Aug 19, 2026
42 checks passed
@vishal-bala
vishal-bala deleted the tests/gate-azure-vectorizer-tests-add-mocked-unit-tests branch August 19, 2026 15:02
@applied-ai-release-bot

Copy link
Copy Markdown

🚀 PR was released in v0.26.0 🚀

@applied-ai-release-bot applied-ai-release-bot Bot added the released This issue/pull request has been released. label Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto:patch Increment the patch version when merged released This issue/pull request has been released.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant