feat: add LinkContentFetcher component for robust URL content retriev…

[iamyuviii]

fix: Fix LinkContentFetcher user-agent rotation race condition

Related Issues

• fixes LinkContentFetcher's User-Agent rotation on retry is scrambled across concurrent multi-URL fetches #12287 12287

Proposed Changes:

Bug:
The LinkContentFetcher tracked its rotating user-agent index (current_user_agent_idx) as an instance variable. When multiple URLs were fetched concurrently (e.g. via ThreadPoolExecutor or asyncio.gather()), the shared state caused a race condition where concurrent requests continuously overwrote the shared index. Furthermore, a finally block in the fetch method forcefully reset the index back to 0, which caused the vast majority of retry attempts across concurrent tasks to incorrectly stick with the unrotated, default agent-0.

Solution:

• Removed the current_user_agent_idx instance state entirely.
• Refactored _get_response and _get_response_async to track their user_agent_idx internally using local method variables. This correctly scopes the rotation logic strictly to each individual request.
• Removed the dependency on tenacity for the synchronous retry logic. By implementing a manual retry loop for the sync path (which functionally mirrors the pre-existing logic in the async path), it resolves the race condition without needing complex context-passing. It additionally eliminates a discrepancy where the tenacity implementation was previously providing one fewer retry attempt than the async implementation.

How did you test it?

Tested manually via a reproduction script utilizing concurrent requests, verifying that independently-failing concurrently fetched URLs now successfully and reliably rotate to agent-1 on their retries instead of incorrectly reusing agent-0.

Notes for the reviewer

None.

Checklist

• I have read the contributors guidelines and the code of conduct.
• I have updated the related issue with new insights and changes.
• I have added unit tests and updated the docstrings.
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
• I have documented my code.
• I have added a release note file, following the contributors guidelines.
• I have run pre-commit hooks and fixed any issue.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

Yuvraj Khichi seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[vercel]

Someone is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[HaystackBot]

Hi @iamyuviii, thanks a lot for your contribution! 🙏

We noticed that the Contributor License Agreement (CLA) check (license/cla) hasn't passed yet, so we've temporarily moved this PR to draft and paused the review assignment.

To get your PR reviewed, please sign the CLA via the link in the license/cla check below (or in the CLA bot comment). As soon as the check turns green, this PR will automatically be marked ready for review again and a reviewer will be re-assigned.

[julian-risch]

Hi @iamyuviii , the problem here seems to be your first commit in this branch 4d69b53 which was not linked to your GitHub account and therefore the CLA check failed.