docs CI: revert search-mode filtering + harden flaky external-service tests#467
Merged
Conversation
…network reset /llms.txt 301-redirects to weaviate.io/llms.txt. Add weaviate.io to the Claude web_fetch allowed_domains so it can follow the redirect, and make test_llms_txt_accessible use a browser UA and treat a transient connection reset/timeout as an inconclusive skip (matching test_llms_txt_code.py) rather than a hard failure.
…ment" This reverts commit 7d55f67.
The generative/hybrid search snippet tests and the Claude llms.txt fetch test intermittently red the suite on transient blips (gRPC UNAVAILABLE, deadline exceeded, read timeouts, 429/5xx) against the shared test cluster + LLM APIs. Add a narrow transient-error retry (utils.retry_on_transient) around the Python, TypeScript, and C# snippet runners, and retry the Claude fetch up to 3x. Real failures (assertion/logic/syntax errors) still fail immediately — only messages matching a small transient-marker set are retried.
- Retry transient Weaviate Cloud connection blips in the agents runner (wrap execute_py_script_as_module; add connection-error transient markers). - test_claude_can_fetch_llms_txt: fix stale section assertion (llms.txt no longer has 'agents'/'cloud' top-level sections -> check 'quickstart' + 'the weaviate stack'), bump max_tokens 2048->4096 to avoid truncated responses. - test_chatgpt_can_search_code_tabs: web_search paraphrases code, so verify the vectorizer is identified (normalized text2vec-weaviate) instead of requiring verbatim lines, and drop the vectorizer hint from the prompt so the check actually proves the code tabs were read from the site.
![orca-security-eu[bot]](https://avatars.githubusercontent.com/in/276244?s=60&v=4){kind=small}
There was a problem hiding this comment.
Orca Security Scan Summary
| Status | Check | Issues by priority | |
|---|---|---|---|
 Passed |
Infrastructure as Code |  0  0  0  0 |
View in Orca |
| Passed |
SAST | 0 0 0 0 |
View in Orca |
| Passed |
Secrets | 0 0 0 0 |
View in Orca |
| Passed |
Vulnerabilities | 0 0 0 0 |
View in Orca |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #464 (already merged). Full docs test suite is green on
testing-ci.Changes
Revert the Query Agent search-mode
filteringrequirement. #464 documentedfilteringas required and addedfiltering="recall"to every search-mode snippet. The Query Agent service no longer requires it (confirmed against the live service —qa.search(...)withoutfilteringno longer 422s), so this reverts the guide wording (back to "recall (default)") and the snippet changes across Python + TypeScript.Transient-retry hardening for flaky external-service snippet tests. The generative/hybrid search tests and the agents tests intermittently red the suite on transient blips against the shared test cluster + LLM APIs. Added
utils.retry_on_transient(narrow marker set: gRPCUNAVAILABLE, deadline exceeded, read timeouts, connection resets, 429/5xx, "overloaded", Weaviate Cloud connection errors) around the Python / TypeScript / C# / agents runners. Real failures (assertion/logic/syntax) still fail immediately — only transient error messages are retried.Indexability test fixes.
test_llms_txt_accessible+test_claude_can_fetch_llms_txt:/llms.txt301-redirects toweaviate.io; allow that domain for the fetch, use a browser UA, and treat a transient network reset as an inconclusive skip.test_claude_can_fetch_llms_txt: the assertedagents/cloudsections no longer exist as top-level llms.txt headings — assert on the actual stable sections (quickstart,the weaviate stack); bumpmax_tokensto avoid truncated responses.test_chatgpt_can_search_code_tabs:web_searchparaphrases code, so verify ChatGPT identifies thetext2vec-weaviatevectorizer (normalized) instead of requiring verbatim lines, and remove the vectorizer hint from the prompt so the check proves the code tabs were actually read from the site. (Verbatim per-language extraction stays hard-verified bytest_claude_can_fetch_code_tabsvia directweb_fetch.)Verification
All suites green on
testing-ci: Docs Code Tests (Agents 27, Python 96, TypeScript 59, Java 37, C# 36), llms.txt (3), Indexability (7 + 63). The ChatGPT test was also run locally to confirm it passes by reading the live page.