Web Scraper API

Web Scraper API for extracting HTML, text and JSON from any URL. One endpoint, any site, no parser per target. This repo has a free web scraper you can run right now, and a universal scraping API that returns the raw page, readable text, or auto-detected JSON.

Last updated: 2026-07-17. Verified against the live API on 2026-07-17.

The full captured responses behind every JSON block on this page are committed in web_scraper_api_data/, and every code example is runnable from web_scraper_api_codes/.

pip install requests
export CHOCODATA_API_KEY="your_key"     # free: 1,000 requests, one-time, no card
python web_scraper_api_codes/fetch_json.py

Those three lines point one endpoint at api.github.com and return this, complete and uncut. We never said the target was JSON: parse=auto worked it out from the body and handed back a parsed object rather than a string.

{
  "url": "https://api.github.com/repos/python/cpython/languages",
  "content_type": "json",
  "json": {
    "Python": 41320114, "C": 23230018, "C++": 401786, "M4": 284155,
    "HTML": 187601, "JavaScript": 102168, "Shell": 86186, "Batchfile": 80254,
    "CSS": 68838, "Makefile": 37389, "Objective-C": 34436, "Roff": 28928,
    "PLSQL": 22886, "PowerShell": 20323, "Rich Text Format": 6905, "Kotlin": 4430,
    "Assembly": 3912, "DTrace": 2017, "TypeScript": 1290, "XSLT": 1174,
    "CMake": 419, "Ruby": 275, "VBScript": 70
  }
}

Point the same endpoint at a web page and content_type comes back html instead. Six requests, three content types, no per-site code:

That is the whole point of this repo. The rest of this page is how it works.

Free web scraper
Avoid getting blocked when web scraping
- Using the Chocodata Web Scraper API
Web Scraper API reference
- Quickstart · Authentication · Parameters · Response shape · Errors · Rate limits and concurrency
- 1. HTML · 2. JSON · 3. Text · 4. Location targeting
Crawl a list of URLs and parse each one
Measured latency

Free web scraper

Most of the web is not defended. If your target is one of those, a plain HTTP client and a parser will do the job:

pip install requests beautifulsoup4
python free_scraper/free_web_scraper.py https://en.wikipedia.org/wiki/Web_scraping
python free_scraper/free_web_scraper.py https://api.github.com/repos/python/cpython

Source: free_scraper/free_web_scraper.py. It fetches with plain requests, auto-detects a JSON body the same way the API does, extracts readable text with BeautifulSoup, and tells you when it got nothing. It parses first and only calls something blocked when the data is genuinely absent, because a working page can still ship the word captcha in its own scripts.

After running the command, your terminal should look something like this:

Wikipedia hands it 229,361 characters without complaint. eBay returns HTTP 403 and 1,832 characters titled Error Page | eBay. Both are real runs from the same machine, minutes apart, and the next section is about the gap between them.

Avoid getting blocked when web scraping

A fetch that comes back empty often has nothing to do with your code. Some sites refuse a stock HTTP client outright, and eBay is one of them.

Point this endpoint at the URL the free script could not read, from the same machine and the same session, and the search page comes back. The signal is parsed out of the body rather than grepped for, so a listing only counts when it is genuinely in the response (evidence):

curl "https://api.chocodata.com/api/v1/universal?api_key=YOUR_KEY&url=https%3A%2F%2Fwww.ebay.com%2Fsch%2Fi.html%3F_nkw%3Dlaptop"

Target	Status	Response	`<title>`	`/itm/` listing URLs found
`ebay.com` search	`200`	678,853 chars	`Laptop for sale \| eBay`	131

131 item URLs, parsed out of the real page, on a target that answers a plain requests call with an error page. Counts are raw pattern matches on a live page and drift by a few between runs; what does not drift is that the listings are there to count. Sites like this one are why this endpoint exists.

What bites you	Why	What it costs you
HTTP 200 is not success	Reddit serves its verification shell with a `200`. A naive scraper parses it, finds nothing, and logs "0 results" rather than "blocked".	The expensive failure is not a crash. It is three weeks of empty data that looked like "no results".
It is not a header problem	eBay refuses the connection itself, not the `User-Agent`. No combination of headers makes a stock HTTP client look like a browser.	You cannot patch your way out in code. It is an infrastructure problem.
Datacenter IPs are pre-blocked	AWS, GCP and Azure ranges are well known. This is why a scraper that works on your laptop dies the moment it runs in CI.	Clean IP supply is a recurring bill, not a one-off fix.
Per-geo content	Sites localise price, currency, language and catalogue to the IP they see. Uncontrolled egress returns a different answer per call.	Data you cannot compare to itself, which is fatal for pricing work.
The markup moves	eBay's `s-item__title` class, which every tutorial still uses, returns 0 matches on the live page today. We hit this while writing this README.	Ongoing maintenance, plus alerting smart enough to tell "empty" from "broken".

Using the Chocodata Web Scraper API

The managed option, and the one this repo is built around. One GET for any URL, html / text / auto-detected json, country-level location targeting, and no proxy management. Free for the first 1,000 requests, one-time, no card: chocodata.com.

It is the option for sites that refuse a stock HTTP client. Chocodata also has around 30 prebuilt endpoints that return parsed, structured JSON for specific sites (search results, product pages, reviews and so on). This endpoint does not parse anything: it hands you the raw page. If a prebuilt endpoint exists for your target, use that instead and skip writing a parser.

Web Scraper API reference

Below is the Web Scraper API reference to get you started: authentication, parameters, response shape, errors, and a worked example for each mode.

Quickstart

curl "https://api.chocodata.com/api/v1/universal?api_key=YOUR_KEY&url=https%3A%2F%2Fexample.com"

import requests

r = requests.get(
    "https://api.chocodata.com/api/v1/universal",
    params={"api_key": "YOUR_KEY", "url": "https://example.com"},
    timeout=90,
)
d = r.json()
print(d["content_type"], len(d["html"]))
# html 559

Get a key at chocodata.com (1,000 requests, one-time, no card).

Authentication

Pass api_key as a query parameter on every request. That is the whole auth model. There is no header, no OAuth, no session.

Parameters

Param	Type	Required	Default	Description
`api_key`	string	yes	-	Your Chocodata API key.
`url`	string (URL)	yes	-	The absolute URL to fetch. Must be a valid URL: `not-a-url` is rejected with a `400`.
`parse`	enum	no	`auto`	One of `auto`, `html`, `text`, `json`. See Response shape.
`country`	string (ISO-2)	no	-	Two-letter country code deciding where the request comes from. Exactly 2 characters or you get a `400`.

That is the complete list.

Response shape

Three fields, always. The payload key follows content_type:

{ "url": "https://example.com", "content_type": "html", "text|html|json": "..." }

The payload is the response the server sends. Content that a page builds client-side after load is not present in it, and parse chooses how that body is handed back to you rather than how it is produced:

`parse`	On an HTML page	On a JSON body
`auto` (default)	`content_type: html`, payload in `html`	`content_type: json`, payload parsed into `json`
`html`	`content_type: html`	`content_type: html`, the raw JSON text unparsed
`text`	`content_type: text`, tags stripped	`content_type: text`
`json`	`content_type: html` (falls back, no error)	`content_type: json`

Branch on content_type, never on what you asked for. parse=json against a non-JSON page does not fail: it silently returns html instead. parse=auto decides by looking at whether the body starts with { or [, so a JSON API that returns a bare string or number will come back as html.

Useful response headers: X-Request-Id, and X-Concurrency-Used / X-Concurrency-Limit / X-Concurrency-Remaining.

Errors

Real captured error bodies, not paraphrases. Nothing below is billed: you are only charged on a 2xx.

Status	`error` code	Meaning	Billed	What to do
`400`	`invalid_params`	`url` missing or not a URL, `parse` not in the enum, `country` not exactly 2 chars. Body lists the exact `path`.	no	Fix the query string.
`401`	`INVALID_API_KEY`	Key missing, unrecognised, or revoked.	no	Check `api_key`. Get one at chocodata.com.
`402`	`INSUFFICIENT_CREDITS`	Balance exhausted.	no	Top up or upgrade at chocodata.com.
`404`	`item_not_found`	The target returned 404. `retryable: false`.	no	Fix the URL. Retrying will not help.
`429`	`RATE_LIMITED`	Over 120 requests/60s, or over your plan's concurrency.	no	Back off; see Rate limits.
`502`	`target_unreachable`	The target refused every attempt. `retryable: true`.	no	Retry.

Two response shapes exist: auth and billing errors nest under error.code (uppercase), while scrape-layer errors are flat with a lowercase error string, and carry a retryable flag when the failure happened at fetch time. Both shapes are shown below.

The scripts in this repo map every documented status onto an actionable message, so a typo'd key does not hand you a stack trace:

A bad key, verbatim (sample):

{"error":{"code":"INVALID_API_KEY","message":"Api key not recognised."}}

A malformed url, verbatim. Note it names the failing path, which makes it machine-readable (sample):

{"error":"invalid_params","issues":[{"validation":"url","code":"invalid_string","message":"Invalid url","path":["url"]}]}

Rate limits and concurrency

Two limits apply at once, and the one that binds first is usually the rate limit, not the concurrency cap.

Rate limit: 120 requests per 60 seconds, per key (sliding window). That is a hard ceiling of 2 requests/second sustained, whatever your plan.

Concurrency: how many requests you may have in flight at once, which varies by plan:

Plan	Concurrent requests
Free	10
Vibe	30
Pro	50
Custom	100 to 500+

Exceed either and you get 429, not a queue. Every request is a synchronous GET: there is no webhook, callback, or async job to poll. Each costs 5 credits (= 1 request), billed only on success.

Sizing, and note which limit actually binds. At Pro you get 50 concurrent slots, and at a ~1.5s median those slots could in principle turn over 30 requests/second. They cannot: the 120/60s rate limit caps you at 2 requests/second first. So plan against 120 requests per minute, or 7,200 an hour, and size your backlog against that rather than against your concurrency number. Keep your pool at or under your plan's concurrency and pace it to the rate limit:

from concurrent.futures import ThreadPoolExecutor
import requests

def one(url):
    r = requests.get("https://api.chocodata.com/api/v1/universal",
                     params={"api_key": KEY, "url": url}, timeout=90)
    return url, (r.json() if r.ok else None)

urls = ["https://example.com", "https://api.github.com/repos/python/cpython"]
with ThreadPoolExecutor(max_workers=10) as pool:   # <= your plan's concurrency
    for url, data in pool.map(one, urls):
        print(url, data and data["content_type"])

1. HTML: raw markup from any URL

The default. Returns the exact markup the server sent, uncut, in html.

Param	Type	Required	Default	Description
`url`	string (URL)	yes	-	Absolute URL of the page.
`parse`	enum	no	`auto`	Force with `parse=html` if the target returns JSON but you want the raw text.

curl "https://api.chocodata.com/api/v1/universal?api_key=YOUR_KEY&url=https%3A%2F%2Fexample.com"

Real response, complete and uncut, all 3 fields verbatim (full sample):

{
  "url": "https://example.com",
  "content_type": "html",
  "html": "<!doctype html><html lang=\"en\"><head><title>Example Domain</title>..."
}

html is a string, not a DOM. You still need a parser (BeautifulSoup, lxml, selectolax) on your side: this endpoint gets you the bytes, not the fields. The 559-character body above is the entire real page, truncated in this block only for display; the committed sample has all of it.

Runnable: web_scraper_api_codes/fetch_html.py

2. JSON: auto-detected API responses

When the body parses as JSON, you get it as parsed JSON under json, not as a string you have to json.loads yourself. You do not have to tell it: parse=auto detects it.

Param	Type	Required	Default	Description
`url`	string (URL)	yes	-	Absolute URL of the JSON endpoint.
`parse`	enum	no	`auto`	`auto` detects. `json` forces an attempt, and falls back to `html` if it does not parse.

curl "https://api.chocodata.com/api/v1/universal?api_key=YOUR_KEY&url=https%3A%2F%2Fapi.github.com%2Frepos%2Fpython%2Fcpython%2Flanguages"

Real response, complete and uncut, all 23 fields verbatim, nothing trimmed (full sample):

{
  "url": "https://api.github.com/repos/python/cpython/languages",
  "content_type": "json",
  "json": {
    "Python": 41320114, "C": 23230018, "C++": 401786, "M4": 284155,
    "HTML": 187601, "JavaScript": 102168, "Shell": 86186, "Batchfile": 80254,
    "CSS": 68838, "Makefile": 37389, "Objective-C": 34436, "Roff": 28928,
    "PLSQL": 22886, "PowerShell": 20323, "Rich Text Format": 6905, "Kotlin": 4430,
    "Assembly": 3912, "DTrace": 2017, "TypeScript": 1290, "XSLT": 1174,
    "CMake": 419, "Ruby": 275, "VBScript": 70
  }
}

json is a parsed object, not a string, so you do not json.loads it yourself. The whole response body above is 446 bytes: check it with curl -s -o /dev/null -w '%{size_download}' "...&url=https%3A%2F%2Fapi.github.com%2Frepos%2Fpython%2Fcpython%2Flanguages".

This is the mode most people miss, and it is the one that makes a mixed crawl simple: a lot of what looks like scraping is really "fetch the JSON endpoint the page itself calls". If you can find that URL in your browser's network tab, you skip HTML parsing entirely and your scraper stops breaking when the markup moves.

Detection is by first character ({ or [) after trimming whitespace, so a JSON body that is a bare string, number or null at the top level comes back as html. Always branch on content_type.

Runnable: web_scraper_api_codes/fetch_json.py

3. Text: readable text extraction

parse=text strips the tags and returns readable text. Useful for search indexing, LLM ingestion, and diffing content without markup churn.

Param	Type	Required	Default	Description
`url`	string (URL)	yes	-	Absolute URL of the page.
`parse`	enum	yes for this mode	`auto`	Must be `text`. `auto` will not choose it for you.

curl "https://api.chocodata.com/api/v1/universal?api_key=YOUR_KEY&url=https%3A%2F%2Fexample.com&parse=text"

Real response, complete and uncut, all 3 fields verbatim (full sample):

{
  "url": "https://example.com",
  "content_type": "text",
  "text": "Example Domain Example Domain This domain is for use in documentation examples without needing permission. Avoid use in operations. Learn more"
}

The 559-character HTML page becomes 142 characters of text. That ratio is the point: you are not moving <div>s into your context window.

It is a tag strip, not a readability extraction. Navigation, cookie banners and footers come through as text alongside the article, and there is no <h1>/<p> structure left to tell them apart. If you need boilerplate removal, run trafilatura or readability over the html mode instead.

4. Location targeting: the same URL from another country

country takes a two-letter code and decides where the request comes from. Sites that localise by location answer differently.

Param	Type	Required	Default	Description
`country`	string (ISO-2)	no	-	e.g. `us`, `de`, `jp`, `br`. Exactly 2 characters.

curl "https://api.chocodata.com/api/v1/universal?api_key=YOUR_KEY&url=https%3A%2F%2Fwww.google.com%2F&country=de"

Real measurement. Same URL, four countries, one minute apart (full sample):

`country`	`<html lang>` served
`us`	`en`
`de`	`de`
`jp`	`ja`
`br`	`pt-BR`

The lang attribute follows the country you asked for, which is how you verify the parameter took effect. This is the parameter to reach for when prices come back in the wrong currency: pin it and the results become comparable to each other.

Targeting is country-level: there is no city, state or ZIP granularity. A code that is two characters but not a real country (country=xx) is not rejected at validation: it fails at fetch time with a 502 target_unreachable, which reads like the site blocked you when actually the country does not exist.

Runnable: web_scraper_api_codes/country_targeting.py

Crawl a list of URLs and parse each one

The reason an any-URL endpoint exists: you have a list, the items are not all the same kind of thing, and you do not want to write a fetcher per site. crawl_url_list.py runs the list concurrently, respects your plan's concurrency, stores each result in a local SQLite dataset you can re-run and diff, and prints what it got:

python web_scraper_api_codes/crawl_url_list.py
python web_scraper_api_codes/crawl_url_list.py my_urls.txt      # one URL per line

That is a real run: five URLs, two content types, 1,138,785 characters, 25 credits. The GitHub row came back as parsed json and the rest as html, and the script never had to know which was which in advance. Export it with one command:

sqlite3 -header -csv crawl.db "select url, content_type, chars, title from pages;" > crawl.csv

The eBay row in that run is the one a stock HTTP client cannot fetch at all, and it arrived through the same code path as example.com. That is what an any-URL endpoint buys you: one fetcher for the whole list.

Measured latency

Real end-to-end wall-clock from a laptop against the live API on 2026-07-17. Includes the upstream fetch and the parse:

Target	Median	Range	n
`example.com` (559 B)	1.3s	0.7 to 1.7s	8
`api.github.com` (JSON)	1.5s	0.7 to 2.6s	8
`google.com` (`country=de`)	1.5s	1.3 to 2.9s	8
`en.wikipedia.org` (241 KB)	2.4s	1.6 to 8.0s	8
`zillow.com` (529 KB)	2.4s	1.7 to 9.0s	8
`ebay.com` (596 KB)	3.3s	2.8 to 3.8s	8

Read the ranges, not just the medians: the 8.0s and 9.0s tails are what your timeout has to survive, which is why every example here uses timeout=90. Small sample (n=8 per row, one machine, one afternoon); reproduce it with the scripts in this repo.

On reliability: eBay returned real items on 12 of 12 consecutive calls in a dedicated run.

License

MIT. See LICENSE.

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
assets		assets
free_scraper		free_scraper
web_scraper_api_codes		web_scraper_api_codes
web_scraper_api_data		web_scraper_api_data
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
requirements.txt		requirements.txt

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Repository files navigation

Web Scraper API

Contents

Free web scraper

Avoid getting blocked when web scraping

Using the Chocodata Web Scraper API

Web Scraper API reference

Quickstart

Authentication

Parameters

Response shape

Errors

Rate limits and concurrency

1. HTML: raw markup from any URL

2. JSON: auto-detected API responses

3. Text: readable text extraction

4. Location targeting: the same URL from another country

Crawl a list of URLs and parse each one

Measured latency

License

About

Uh oh!

Releases

Packages

Uh oh!

Contributors

Uh oh!

Languages

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

Web Scraper API

Contents

Free web scraper

Avoid getting blocked when web scraping

Using the Chocodata Web Scraper API

Web Scraper API reference

Quickstart

Authentication

Parameters

Response shape

Errors

Rate limits and concurrency

1. HTML: raw markup from any URL

2. JSON: auto-detected API responses

3. Text: readable text extraction

4. Location targeting: the same URL from another country

Crawl a list of URLs and parse each one

Measured latency

License

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages