Skip to content

Add ScrapeUnblocker integration - #545

Open
Kontuzijus wants to merge 3 commits into
deepset-ai:mainfrom
ScrapeUnblocker:add-scrapeunblocker-integration
Open

Add ScrapeUnblocker integration#545
Kontuzijus wants to merge 3 commits into
deepset-ai:mainfrom
ScrapeUnblocker:add-scrapeunblocker-integration

Conversation

@Kontuzijus

Copy link
Copy Markdown

Adds an integration page for ScrapeUnblocker.

ScrapeUnblocker renders web pages in a real browser behind anti-bot protections
(Cloudflare, DataDome, PerimeterX, Akamai) and returns raw HTML or AI-parsed
structured JSON, so a pipeline can ingest pages that a plain HTTP fetch cannot
reach.

The scrapeunblocker-haystack package is published and provides two components:

Both code examples in the page were run against the published package before
opening this PR. The logo is added as logos/scrapeunblocker.png.

@Kontuzijus
Kontuzijus requested a review from a team as a code owner July 18, 2026 13:07
@vercel

vercel Bot commented Jul 18, 2026

Copy link
Copy Markdown

@Kontuzijus is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new Haystack integrations page documenting the scrapeunblocker-haystack package, so users can discover and use ScrapeUnblocker components in Haystack 2.x pipelines.

Changes:

  • Adds integration front-matter metadata (logo, links, categorization) for ScrapeUnblocker.
  • Documents installation and basic usage for ScrapeUnblockerFetcher and ScrapeUnblockerWebSearch.
  • Adds an end-to-end pipeline example showing how to fetch a protected page and prompt an LLM with its contents.

馃挕 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +115 to +139
from haystack import Pipeline
from haystack.components.builders import ChatPromptBuilder
from haystack.components.converters import HTMLToDocument
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage

from scrapeunblocker_haystack import ScrapeUnblockerFetcher

prompt = [
ChatMessage.from_user(
"Answer the question using the pages below.\n\n"
"{% for doc in documents %}{{ doc.content }}\n{% endfor %}\n"
"Question: {{ question }}"
)
]

pipe = Pipeline()
pipe.add_component("fetcher", ScrapeUnblockerFetcher())
pipe.add_component("converter", HTMLToDocument())
pipe.add_component("prompt_builder", ChatPromptBuilder(template=prompt, required_variables="*"))
pipe.add_component("llm", OpenAIChatGenerator())

pipe.connect("fetcher.documents", "converter.sources")
pipe.connect("converter.documents", "prompt_builder.documents")
pipe.connect("prompt_builder.prompt", "llm.messages")

@kacperlukawski kacperlukawski left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @Kontuzijus Are you sure the provided examples work, like pointed out by Copilot?

ScrapeUnblockerFetcher outputs List[Document], while HTMLToDocument.sources
expects list[str | Path | ByteStream], so the documented pipeline raised
PipelineConnectError and could never run. Drop the converter and connect
fetcher.documents straight to prompt_builder.documents.

Also document base_url and timeout for ScrapeUnblockerWebSearch.
@Kontuzijus

Copy link
Copy Markdown
Author

Good catch, and thanks for checking - Copilot was right, the pipeline example was broken.

ScrapeUnblockerFetcher outputs List[Document], while HTMLToDocument.sources expects list[str | Path | ByteStream], so that pipeline never even got to run - it failed at connect time:

PipelineConnectError: Cannot connect 'fetcher.documents' with 'converter.sources':
their declared input and output types do not match.
'fetcher':
 - documents: List[Document]
'converter':
 - sources: list[str | Path | ByteStream] (available)

Fixed exactly as Copilot suggested: the converter is gone and fetcher.documents now connects straight to prompt_builder.documents, since the fetcher already emits Document objects.

To make sure I am not guessing a second time, I executed every Python snippet on the page, parsed straight out of the updated scrapeunblocker.md, in a clean venv (haystack-ai 3.0.0, scrapeunblocker-haystack installed from PyPI) against the live API:

  1. ScrapeUnblockerFetcher basic example - 1 Document returned, HTML content, meta url / content_type / parsed_data.
  2. ScrapeUnblockerWebSearch basic example - 5 Documents with title / link / position in meta.
  3. The pipeline example - runs end to end, including OpenAIChatGenerator, and answers the question about the fetched page.

One more docs fix in the same commit: base_url and timeout were missing from the ScrapeUnblockerWebSearch parameter table, although the component accepts both.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants