Skip to content

ChocoData-com/bing-scraper

Repository files navigation

Bing Scraper

Bing Scraper

Bing Scraper for extracting organic search results, titles, links, snippets, images, videos and recipes from Bing.com. This repo has a free Bing web scraping script you can run right now, and a Bing search data API with 4 endpoints returning real structured JSON.

Last updated: 2026-07-16. Working against Bing.com as of July 2026, and re-verified whenever Bing changes their markup.

Every JSON block on this page was captured from the live API on 2026-07-16. Long arrays are trimmed to the first item and each block states exactly what was cut. Full uncut samples are committed in bing_scraper_api_data/, so you can diff this page against them. Every code example calls the actual API and is runnable from bing_scraper_api_codes/.

pip install requests
export CHOCODATA_API_KEY="your_key"     # free: 1,000 requests, one-time, no card
python bing_scraper_api_codes/search.py

Those three lines return this, live from Bing.com:

{
  "position": 1,
  "title": "Machine Learning Tutorial - GeeksforGeeks",
  "link": "https://www.geeksforgeeks.org/machine-learning/machine-learning/",
  "displayed_link": "https://www.geeksforgeeks.org › machine-learning",
  "snippet": "Machine learning is a branch of Artificial …",
  "date": "Jun 5, 2026",
  "source": "bing"
}

...for all 5 organic results the endpoint returned, with 7 fields each:

Retrieved Bing organic search results

That is the whole point of this repo. The rest of this page is how it works and what each endpoint returns.

Bing server-renders its organic results into the page, which is why a general SERP endpoint that returns structured organic results is possible at all here.


Contents


Free Bing Scraper

Bing server-renders its organic results into plain <li class="b_algo"> blocks, so you can extract structured data from Bing search pages without a headless browser or JavaScript rendering. No key, no cost:

pip install requests beautifulsoup4
python free_scraper/bing_free_scraper.py "machine learning"

Source: free_scraper/bing_free_scraper.py. It walks the li.b_algo blocks, decodes Bing's redirect links, and emits position, title, link, displayed_link, snippet, date.

After running the command, your terminal should look something like this:

Free Bing scraper returning real organic results

It works. Run against Bing.com on 2026-07-16 it returned 9 organic results for this query, and across 30 sequential requests on rotating queries every single one came back with a full 10-result organic block, in 6.0 seconds total, with no bot check, no challenge page and no rate limiting. Bing is the one major search engine that still answers a plain HTTP request with a server-rendered SERP.

There is one trap, and it is the exact opposite of what scraping experience teaches you. The next section is about that.

Avoid getting blocked when scraping Bing

Mostly, you do not get blocked. We ran the script above 30 times back to back from a clean residential IP while writing this README: 30 of 30 returned 10 organic results, in 6.0 seconds, with no challenge page and no 429.

What actually costs you data on Bing is subtler, and the first one is genuinely counterintuitive:

Three ways to ask Bing for the same query

Sending a realistic Chrome User-Agent is what breaks it. Bing serves the server-rendered SERP to simple clients, and the client-side-rendered SERP to anything claiming to be a modern browser, because a modern browser is expected to run the JavaScript. So the single most common scraping instinct, spoofing a real browser, is what returns nothing:

What we measured (q="machine learning", n=6 each) No browser UA Modern Chrome UA
HTTP status 200 200
Response size ~116 to 161 KB ~71 to 72 KB
<title> machine learning - Search machine learning - Search
li.b_algo blocks 9 0
Organic results parsed 9 0
Outcome across all 6 attempts 9 results, 6 of 6 0 results, 6 of 6

Sizes are approximate because Bing's page weight drifts by a few hundred bytes per request; the result counts do not drift at all.

Reproduce it yourself in ten seconds: python free_scraper/bing_free_scraper.py "machine learning" --chrome-ua

The Chrome User-Agent trap: HTTP 200 and zero results

An HTTP 200 with a 72 KB page and zero results is the expensive failure, because it does not look like a failure. It looks like "this query has no results".

Here is the rest of what bites you, and what each one costs:

What bites you Why What it costs you
The Chrome UA trap Bing gives a browser-shaped client the JS shell. Your scraper gets 200 OK and zero rows. Silent empty data. The failure mode is "0 results" in your logs, not an exception.
The SERP binds to your egress IP Bing localises to the country your request exits from. Ours ran from Lithuania and Bing returned a Lithuanian SERP (<title>web scraping - Ieškoti</title>) until we pinned the language. Ranks you cannot compare across machines, CI runners, or teammates. Fatal for rank tracking.
The links are not the links Every organic href is a bing.com/ck/a?...&u=a1<base64url> redirect, not the destination. If you skip the decode step you store 10 Bing redirects instead of 10 URLs, and you find out weeks later.
Bing does refuse some requests Not many, but they exist. A site:github.com scraper query came back 502 with a 49 KB page and no organic block at all, where the same query shape normally parses fine. Your pipeline needs to tell "Bing said no" apart from "no results exist".
The markup moves b_algo and the redirect encoding change periodically. Your parser silently returns []. Ongoing maintenance, plus alerting smart enough to tell "empty" from "broken".
There is no official API to fall back on Microsoft retired the Bing Search APIs. See below. When your scraper breaks there is no supported paid escape hatch to switch to.

The official API is gone. From Microsoft's lifecycle announcement, dated 2025-05-15:

"Bing Search APIs will be retired on August 11, 2025. Any existing instances of Bing Search APIs will be decommissioned completely."

Source: Bing Search APIs Retiring on August 11, 2025, Microsoft Learn. The old Bing Web Search API documentation now carries is_retired: true and redirects into Microsoft's /previous-versions/ archive.

Microsoft did not leave you with nothing. The announcement points customers to Grounding with Bing Search as part of Azure AI Agents, which, in Microsoft's words, "allows Azure AI Agents to incorporate real-time public web data when generating responses with an LLM". That is a different product from a SERP API. It feeds web context into an LLM answer. It does not hand you ranked organic results as structured rows you can store, diff and chart. If what you need is position 1 through 5 with titles, links and snippets, the officially supported path to it no longer exists. That is the gap this repo is in.


Using the Chocodata Bing Scraper API

The Chocodata Bing Scraper API is the managed option, and the one this repo is built around. Four endpoints for Bing search data extraction at scale (organic results, images, videos and recipes), parsed JSON instead of HTML, and no proxy management. Free for the first 1,000 requests.

What the API adds:

  • Images, videos and recipes. The free script does organic results only. Three of the four endpoints below have no free equivalent here, and building them means parsing Bing's iusc image payloads and mc_vtvc video cards yourself. This is why most people end up on the API.
  • A pinned market. country fixes the SERP to a market instead of letting it bind to whatever IP your code happens to run from. This is the difference between rank data you can compare over time and rank data you cannot, and the free script cannot do it.
  • It runs from anywhere. Your CI runner does not need a clean IP supply, and the request shape that gets you a server-rendered SERP is our problem to keep working, not yours.
  • Somebody else maintains the parser. When b_algo or the redirect encoding moves, that is our on-call, not yours.

Chocodata's platform benchmark is a ~99% success rate; the measured Bing numbers are in Measured latency.


Bing Scraper API reference

Below is the Bing Scraper API reference to get you started: four endpoints, each with its parameters, a curl you can paste, and the JSON it returns.

Quickstart

curl "https://api.chocodata.com/api/v1/bing/search?api_key=YOUR_KEY&q=machine%20learning"
import requests

r = requests.get(
    "https://api.chocodata.com/api/v1/bing/search",
    params={"api_key": "YOUR_KEY", "q": "machine learning"},
    timeout=90,
)
top = r.json()["organic_results"][0]
print(top["position"], top["title"], top["link"])
# 1 Machine Learning Tutorial - GeeksforGeeks https://www.geeksforgeeks.org/machine-learning/machine-learning/

After running the command, your terminal should look something like this:

Running the Bing Scraper API search endpoint

Get a key at chocodata.com (1,000 requests, one-time, no card).

Authentication

Pass api_key as a query parameter on every request. That is the whole auth model.

Global parameters

Accepted by every endpoint below:

Param Type Required Default Description
api_key string yes - Your Chocodata API key.
q string yes - The search query. Must be at least 1 character.
count int no varies Upper bound on returned rows. See each endpoint: Bing decides how many it renders, so this caps the list, it does not guarantee it.

country (ISO-2) is accepted by search, images and videos. It sets the Bing market (mkt=en-XX), which pins the SERP to a market instead of letting it bind to your egress location. recipe does not take it.

Each request costs 5 credits (= 1 request). Responses are billed only on success (2xx).

Errors

Real captured error bodies, not paraphrases. Nothing below is billed: you are only charged on a 2xx.

Status error code Meaning Billed What to do
400 invalid_params A required param is missing or out of range. Body lists the exact path and constraint. no Fix the query string.
401 INVALID_API_KEY Key missing, unrecognised, or revoked. no Check api_key. Get one at chocodata.com.
402 INSUFFICIENT_CREDITS Balance exhausted. no Top up ($0.90 / 1,000 requests, never expires) or upgrade.
429 RATE_LIMITED Over your plan's concurrency. no Back off and retry; see Rate limits.
502 extraction_failed Bing did not return a SERP we could parse for this request. no Retry.

Two response shapes exist: auth/billing errors nest under error.code (uppercase), while scrape-layer errors are flat with a lowercase error string. Both are shown below.

The scripts in this repo map every documented status onto an actionable message, so a typo'd key does not hand you a stack trace:

Bing Scraper API error handling

A bad key, verbatim:

curl "https://api.chocodata.com/api/v1/bing/search?api_key=totally_invalid_key_123&q=machine%20learning"
{"error":{"code":"INVALID_API_KEY","message":"Api key not recognised."}}

Omitting the key entirely gives you the hint instead, verbatim:

{"error":{"code":"INVALID_API_KEY","message":"Missing api_key query parameter.","hint":"Pass your key as ?api_key=<your_key> in the request URL."}}

A bad parameter, verbatim. It names the exact field and the constraint it violated, which is why the scripts can print 400 invalid_params: count: Number must be less than or equal to 50 instead of a traceback:

{"error":"invalid_params","issues":[{"code":"too_big","maximum":50,"type":"number","inclusive":true,"exact":false,"message":"Number must be less than or equal to 50","path":["count"]}]}

Rate limits and concurrency

There is no per-minute request cap. The limit is concurrency: how many requests you may have in flight at once.

Plan Concurrent requests
Free 10
Vibe 30
Pro 50
Custom 100 to 500+

Exceed it and you get 429, not a queue. Every endpoint is a synchronous GET: there is no webhook, callback, or async job to poll. Bing is fast (a ~1.9s median, see Measured latency), but the tail runs to ~8.7s, which is why the examples use timeout=90.

Sizing: at Pro (50 concurrent) and a ~1.9s median search, one worker pool sustains roughly 50 / 1.9 = 26 requests/second. Fan out with a thread pool:

from concurrent.futures import ThreadPoolExecutor
import requests

def one(q):
    r = requests.get("https://api.chocodata.com/api/v1/bing/search",
                     params={"api_key": KEY, "q": q}, timeout=90)
    return q, (r.json().get("organic_results", []) if r.ok else [])

queries = ["web scraping", "python tutorial", "seo tools", "rank tracking", "serp api"]
with ThreadPoolExecutor(max_workers=10) as pool:   # <= your plan's concurrency
    for q, results in pool.map(one, queries):
        print(q, len(results))

1. Search: organic results, titles, links and snippets

Ranked Bing organic search results for a keyword, with position, title, decoded link, displayed URL, snippet and article date.

Param Type Required Default Description
q string yes - Search keywords.
count int (1-50) no 10 Upper bound on rows. Bing decides how many it server-renders, so this caps the list rather than guaranteeing it.
country string (ISO-2) no us Bing market (mkt=en-XX). Pin this for reproducible ranks.
first int (>=1) no 1 Result offset. Accepted and echoed; Bing returned the same first-page links at first=11 and first=31.
curl "https://api.chocodata.com/api/v1/bing/search?api_key=YOUR_KEY&q=machine%20learning"

Real response. organic_results is cut to 1 of 5; the result object itself is complete, all 7 fields verbatim (full sample):

{
  "query": "machine learning",
  "engine": "bing",
  "results_count": 5,
  "organic_results": [
    {
      "position": 1,
      "title": "Machine Learning Tutorial - GeeksforGeeks",
      "link": "https://www.geeksforgeeks.org/machine-learning/machine-learning/",
      "displayed_link": "https://www.geeksforgeeks.org › machine-learning",
      "snippet": "Machine learning is a branch of Artificial …",
      "date": "Jun 5, 2026",
      "source": "bing"
    }
  ]
}

link is the field most people come for, and it is the one a DIY scraper usually gets wrong: Bing serves every organic href as a bing.com/ck/a?...&u=a1<base64url> redirect, and it has to be base64url-decoded back to the destination or you store a Bing tracking URL instead of geeksforgeeks.org/.... date is split out of the snippet rather than left glued to the front of it (Bing renders it as Jun 5, 2026 · Machine learning is...), and it is null on undated results: in this very sample, positions 2 and 3 (Wikipedia and IBM) both have "date": null, which is the null-handling you have to code against. displayed_link is Bing's breadcrumb rendering, chevrons and all, not a URL: parse link instead. source is a constant "bing" on every row, present so responses stay uniform if you merge engines; it is not per-result provenance.

What to expect from the response, measured over 20 distinct queries on 2026-07-16:

  • 4 to 7 organic rows per call. The distribution was 5 rows on 17 of 20 calls, 7 on two, 4 on one. count is an upper bound that caps our list; it does not make Bing server-render more. Setting count=50 still returned 5.
  • first does not paginate. It is accepted and echoed, but at first=11 and first=31 the organic block came back with the same links as first=1 (5 of 5 identical). Build against the first page.
  • Organic results are what Bing server-renders here. The schema can carry ads, knowledge_graph, related_questions, answer_box and related_searches, but Bing did not server-render any of them for this egress on any query we tried, including ones picked to trigger them (how tall is mount everest, who is elon musk, buy running shoes).

Reliability: 20 of 20 calls returned a parseable SERP in the run above.

Runnable: bing_scraper_api_codes/search.py


2. Images: image URLs, thumbnails and source pages

Bing image search: the full-size image URL, Bing's thumbnail, and the page each image was found on.

Param Type Required Default Description
q string yes - Search keywords.
count int (1-200) no 50 Upper bound on rows.
country string (ISO-2) no us Bing market (mkt=en-XX).
safe_search strict / moderate / off no strict SafeSearch level. Defaults to strict.
first int (>=1) no 1 Result offset. Accepted and echoed; first=31 returned 19 of the same 20 images as first=1.
curl "https://api.chocodata.com/api/v1/bing/images?api_key=YOUR_KEY&q=golden%20retriever&count=20"

Real response. results is cut to 1 of 20; the result object is complete, all 7 fields verbatim (full sample):

{
  "query": "golden retriever",
  "page": 1,
  "results_count": 20,
  "total_results": 20,
  "results": [
    {
      "position": 1,
      "id": "cdn.pixabay.com/photo/2024/07/09/17/36/golden-retriever-8884127_1280.jpg",
      "title": "40,000+ Free Baby Golden Retriever & Golden Retriever Images - Pixabay",
      "image": "https://cdn.pixabay.com/photo/2024/07/09/17/36/golden-retriever-8884127_1280.jpg",
      "thumbnail": "https://ts2.explicit.bing.net/th?id=OIP.C-nxeb5qnLQJiCMPDui4RwHaLH&pid=15.1",
      "source_page": "https://pixabay.com/images/search/baby golden retriever/",
      "source": "pixabay.com"
    }
  ]
}

Bing images endpoint output

image is the full-size original on the publisher's own host, which is the field that makes this useful and the one you cannot get from the thumbnail grid: thumbnail points at Bing's CDN and will rot, image points at the source. source_page is where it was found, source is that page's host, pre-parsed for attribution or filtering. This is the most complete of the four endpoints: on the committed 20-row capture, all 7 fields are populated on all 20 rows, across 15 distinct source domains. Diff it yourself, it is right there in the repo.

Note source_page above contains literal spaces (.../search/baby golden retriever/). That is what Bing emits, and we pass it through verbatim rather than silently re-encoding it. Run it through urllib.parse.quote before you fetch it.

The row count is not deterministic at any count, because Bing varies how much of the image grid it server-renders per request. Same query, same parameters, measured 2026-07-16:

Requested n What actually came back
count=20 8 20 rows on 6 calls, 12 rows on 2
count=50 8 12 rows on 5 calls, 30 rows on 3 (and 48 on an earlier call)

Treat count as a cap and always read results_count, never assume you got what you asked for. If your pipeline needs exactly N images, request more than N and slice. first does not paginate here either: first=31 came back with 19 of the same 20 images as first=1.

Runnable: bing_scraper_api_codes/images.py


3. Videos: titles, channels, durations and view counts

Bing video search, with the metadata Bing renders on each card: duration, channel, publish date and view count.

Param Type Required Default Description
q string yes - Search keywords.
count int (1-100) no 50 Upper bound on rows.
country string (ISO-2) no us Bing market (mkt=en-XX).
curl "https://api.chocodata.com/api/v1/bing/videos?api_key=YOUR_KEY&q=python%20tutorial&count=20"

Real response. results is cut to 1 of 20; the result object is complete, all 9 fields verbatim (full sample):

{
  "query": "python tutorial",
  "results_count": 20,
  "results": [
    {
      "position": 1,
      "title": "How Python Actually Works | Python Explained Simply | Intellipaat",
      "link": "https://www.youtube.com/watch?v=IQcV6yz_jB4",
      "thumbnail": "https://th.bing.com/th/id/OVP.RMiE6S_xw4CSkBTBnoGAhADKFo?w=196&h=110&c=7&rs=1&qlt=70&o=7&pid=2.1&rm=3",
      "duration": "1:10",
      "source": "YouTube",
      "channel": "Intellipaat",
      "published": "1 month ago",
      "views": "8.3K views"
    }
  ]
}

Bing videos endpoint output

link resolves to the real destination (a youtube.com/watch?v=... URL here), not a Bing player page, which is what makes this usable as a discovery feed. This is the richest of the four endpoints at 9 fields, and it aggregates across hosts, so source tells you where each result actually lives. Worth knowing before you build on it: Bing's video mix is not all YouTube. On the committed 20-row capture, source is YouTube on 12 rows and TikTok on 8. If your pipeline assumes YouTube URLs, 40% of this sample would break it.

views and published are strings exactly as Bing renders them ("8.3K views", "1 month ago"), not numbers and not timestamps. They pass through unparsed: "8.3K" has already lost precision at the source, so parse them yourself if you need them numeric and know what you are inferring. Row count is not deterministic either: the same query at count=20 returned 20 rows on two calls, 18 on three, and 10 on one, so treat count as a cap and read results_count.

Runnable: bing_scraper_api_codes/videos.py


4. Recipe (beta): recipe titles, sources and links

Bing recipe results. This endpoint is beta.

Param Type Required Default Description
q string yes - Recipe keywords. "recipe" is appended if your query does not contain it.
count int (1-60) no 30 Upper bound on rows.
curl "https://api.chocodata.com/api/v1/bing/recipe?api_key=YOUR_KEY&q=banana%20bread"

Real response. results is cut to 1 of 5; the result object is complete, all 7 fields verbatim (full sample):

{
  "query": "banana bread",
  "results_count": 5,
  "block": "organic_fallback",
  "results": [
    {
      "position": 1,
      "title": "Banana Banana Bread Recipe (with Video)",
      "link": "https://www.allrecipes.com/recipe/20144/banana-banana-bread/",
      "source": "allrecipes.com",
      "rating": null,
      "total_time": null,
      "thumbnail": null
    }
  ]
}

rating, total_time and thumbnail come from Bing's recipe carousel, which is query-dependent: it did not render on any of the 6 queries we tested (banana bread, chocolate chip cookies, lasagna, pancakes, chicken curry, apple pie recipe). Every one came back block: "organic_fallback", so the three nulls above are what the fallback returns, not an anomaly of this one capture.

What the fallback gives you is real recipe destinations (allrecipes.com, sallysbakingaddiction.com, simplyrecipes.com), filtered to recipe-relevant rows. When Bing returns nothing recipe-shaped, the endpoint returns a typed miss rather than dumping unrelated organic results on you and calling them recipes.

link and source are the fields to build on: they are populated on every row and they are real recipe destinations. rating and total_time fill in when Bing renders the carousel for your query.

Always branch on block. For a list of recipe URLs for a dish, this works today. Ratings and cook times ride on a carousel Bing renders at its own discretion, which is why the endpoint is marked beta.

Runnable: bing_scraper_api_codes/recipe.py


Track Bing rankings for your keywords over time

Tracking where a domain ranks is the main reason people scrape a search engine, so that use case is in the repo end to end rather than as a snippet. rank_tracker.py checks a domain's Bing position for a list of keywords, stores every observation as a local dataset in SQLite (export it to CSV with one sqlite3 command), and prints what moved since the last run:

python bing_scraper_api_codes/rank_tracker.py wikipedia.org "machine learning" "climate change"
# machine learning                       rank   2  (of 5 shown; #1 is geeksforgeeks.org)
# climate change                         rank   2  (of 5 shown; #1 is science.nasa.gov)
#
# 2 keyword(s) checked for wikipedia.org | 0 movement(s) | 2 observations in bing_ranks.db
# No movement yet. Run it again tomorrow, or schedule it (cron / GitHub Actions).

Bing SERP rank tracker running

That is a real first run: the first pass has nothing to compare against, so it seeds the database and says so. Run it again after ranks move and each change prints as a diff line (UP 4 -> 2, DOWN 2 -> 5, DROPPED OUT (was 3)).

Pin country and keep it pinned, or you are comparing a US SERP to whatever market your CI runner's IP resolved to last night.

Scope: because the search endpoint returns 4 to 7 organic rows with no working pagination, a domain outside the top ~5 reports as - rather than as a numeric rank. This tracks top-5 presence.

One API call per keyword per run. 1,000 free requests covers a year of daily checks on two keywords.


Measured latency

Real end-to-end wall-clock, measured from a laptop against the live API on 2026-07-16. This includes the upstream fetch and the parse. Each row is its own timed run, so the n here is independent of the (larger) samples quoted in the endpoint sections above:

Endpoint Median Range n Success
/bing/search 1.9s 0.8 to 8.7s 20 20/20
/bing/images 2.3s 1.9 to 5.5s 5 5/5
/bing/videos 2.2s 1.5 to 3.4s 5 5/5
/bing/recipe 1.8s 1.2 to 2.3s 5 5/5

Read the ranges, not just the medians. Bing is a fast target and these are the quickest numbers we publish on any endpoint. The search tail at 8.7s is a request that needed re-attempting upstream before it came back with real data.

Small samples (n=5 to 20) from a single day; reproduce them yourself with the scripts here.


License

MIT. See LICENSE.