Add ScrapeUnblocker integration - #545
Conversation
|
@Kontuzijus is attempting to deploy a commit to the deepset Team on Vercel. A member of the Team first needs to authorize it. |
There was a problem hiding this comment.
Pull request overview
Adds a new Haystack integrations page documenting the scrapeunblocker-haystack package, so users can discover and use ScrapeUnblocker components in Haystack 2.x pipelines.
Changes:
- Adds integration front-matter metadata (logo, links, categorization) for ScrapeUnblocker.
- Documents installation and basic usage for
ScrapeUnblockerFetcherandScrapeUnblockerWebSearch. - Adds an end-to-end pipeline example showing how to fetch a protected page and prompt an LLM with its contents.
馃挕 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| from haystack import Pipeline | ||
| from haystack.components.builders import ChatPromptBuilder | ||
| from haystack.components.converters import HTMLToDocument | ||
| from haystack.components.generators.chat import OpenAIChatGenerator | ||
| from haystack.dataclasses import ChatMessage | ||
|
|
||
| from scrapeunblocker_haystack import ScrapeUnblockerFetcher | ||
|
|
||
| prompt = [ | ||
| ChatMessage.from_user( | ||
| "Answer the question using the pages below.\n\n" | ||
| "{% for doc in documents %}{{ doc.content }}\n{% endfor %}\n" | ||
| "Question: {{ question }}" | ||
| ) | ||
| ] | ||
|
|
||
| pipe = Pipeline() | ||
| pipe.add_component("fetcher", ScrapeUnblockerFetcher()) | ||
| pipe.add_component("converter", HTMLToDocument()) | ||
| pipe.add_component("prompt_builder", ChatPromptBuilder(template=prompt, required_variables="*")) | ||
| pipe.add_component("llm", OpenAIChatGenerator()) | ||
|
|
||
| pipe.connect("fetcher.documents", "converter.sources") | ||
| pipe.connect("converter.documents", "prompt_builder.documents") | ||
| pipe.connect("prompt_builder.prompt", "llm.messages") |
kacperlukawski
left a comment
There was a problem hiding this comment.
Thank you, @Kontuzijus Are you sure the provided examples work, like pointed out by Copilot?
ScrapeUnblockerFetcher outputs List[Document], while HTMLToDocument.sources expects list[str | Path | ByteStream], so the documented pipeline raised PipelineConnectError and could never run. Drop the converter and connect fetcher.documents straight to prompt_builder.documents. Also document base_url and timeout for ScrapeUnblockerWebSearch.
|
Good catch, and thanks for checking - Copilot was right, the pipeline example was broken.
Fixed exactly as Copilot suggested: the converter is gone and To make sure I am not guessing a second time, I executed every Python snippet on the page, parsed straight out of the updated
One more docs fix in the same commit: |
Adds an integration page for ScrapeUnblocker.
ScrapeUnblocker renders web pages in a real browser behind anti-bot protections
(Cloudflare, DataDome, PerimeterX, Akamai) and returns raw HTML or AI-parsed
structured JSON, so a pipeline can ingest pages that a plain HTTP fetch cannot
reach.
The
scrapeunblocker-haystackpackage is published and provides two components:ScrapeUnblockerFetcher- fetch URLs, one Document per pageScrapeUnblockerWebSearch- Google organic results as DocumentsPyPI: https://pypi.org/project/scrapeunblocker-haystack
Repo: https://github.com/ScrapeUnblocker/scrapeunblocker-haystack
Both code examples in the page were run against the published package before
opening this PR. The logo is added as
logos/scrapeunblocker.png.