Feature request: opt-in block_internal_urls egress filter for the library layer

[joysinleung]

Thanks for following up from the email thread, @ntohidi.

Quick correction on my side: the issue body initially only contained a literal file path — I mistakenly relied on @path expansion with gh api, which doesn't expand files (that's a curl / --body-file flag). Pasting the real content here.

Alignment

We're aligned on the classification. The library is a user agent invoked by a trusted caller, so destination filtering should stay opt-in, and the SSRF trust boundary remains at the Docker API server where egress_broker.py already enforces it. The agentic / LLM-chosen-URL case is the scenario that justifies exposing the same primitives to library callers.

Proposed design

• Flag: block_internal_urls: bool (default False, opt-in). Set per-crawl so callers who embed Crawl4AI in an agent can opt in without a global change.
• Chokepoint: a single host-validation call inserted right after the existing scheme allow-list check in AsyncCrawlerStrategy.crawl() (crawl4ai/async_crawler_strategy.py). It must cover both egress paths:
  • HTTP path: AsyncHTTPCrawlerStrategy._handle_http() (aiohttp)
  • Browser path: browser_manager.py → Playwright goto()
• Logic reuse: port the existing validate_url_destination + resolve_and_pin (DNS pinning) + per-hop redirect revalidation from deploy/docker/utils.py / egress_broker.py into a shared helper (e.g. crawl4ai/url_safety.py) so the library and the Docker server share one implementation — no duplicated trust logic.
• Blocked ranges: loopback (127.0.0.0/8, ::1), private (10/8, 172.16/12, 192.168/16, fc00::/7), link-local (169.254/16 incl. cloud metadata 169.254.169.254, fe80::/10), and 0.0.0.0/8. The resolved IP must be checked after DNS (pin) and after every redirect hop (revalidate) to prevent DNS-rebinding / redirect-to-internal bypasses.
• Behavior on block: raise a BlockedURL exception (or return a failed CrawlResult with a clear error) rather than fetching.

Open questions (happy to match maintainer preference)

1. Flag name — block_internal_urls vs deny_private_destinations vs egress_filter?
2. Where the chokepoint sits — shared base crawl() vs per-strategy hooks?
3. Browser redirect following — should the redirect revalidation also cover hops taken by Playwright goto, or only the initial URL?

I'd be happy to draft the PR implementing this once we settle the surface.

[SohamKukreti]

Hey @joysinleung, can you share the description of the feature request?

[ntohidi]

Thanks for opening this @joysinleung — following up from our email thread.

Heads up: the issue body didn't render. It currently shows only the literal path @/tmp/crawl4ai_feature_request.md instead of the file contents (the gh issue create -F / --body-file flag is what expands a file; @path isn't expanded in the body). Could you paste the text in directly?

For context, we're aligned on the classification: the library layer is a user agent invoked by a trusted caller, so destination filtering stays opt-in, and the SSRF trust boundary remains at the Docker API server where egress_broker.py already enforces it. The agentic / LLM-chosen-URL case you described is the scenario that justifies exposing the same primitives to library callers.

Once the design is in the body we can discuss the surface (flag name, where the chokepoint sits, redirect revalidation). You mentioned you'd be up for drafting the PR — happy to review.