A small, reusable web scraper that pulls structured data from a public site and writes a clean CSV. Handles pagination, missing fields, and polite request delays.
The kind of tool that eats a repetitive "extract this data from the website" ticket in five minutes instead of five hours.
- Walks paginated pages until there are no more.
- Extracts three fields per record (quote text, author, tags in the demo target).
- Skips malformed cards gracefully rather than crashing the whole run.
- Sleeps politely between requests so it plays nice with the target server.
- Writes results to a single CSV with headers.
The demo target is quotes.toscrape.com — a public sandbox site meant for scraper testing. Swap the selectors and base URL for whatever you actually need to scrape.
git clone https://github.com/weli-dev/web-scraper-py
cd web-scraper-py
pip install -r requirements.txt
python scraper.pyExpected output:
Scraping page 1 ...
Scraping page 2 ...
...
Done. 100 records written to quotes.csv
A sample quotes.csv from the demo run is included so you can see the output shape without executing anything.
The demo run produces a CSV like:
| quote | author | tags |
|---|---|---|
| "The world as we have created it..." | Albert Einstein | change, deep-thoughts, thinking, world |
| "It is our choices, Harry..." | J.K. Rowling | abilities, choices |
| ... | ... | ... |
- Web-Scraper-Deliverable.pdf — one-pager describing scope, approach, and result.
- screenshots/ — captures of the tool running end-to-end.
I build AI-agent and automation systems — Claude Code / MCP integrations, LLM-into-workflow tooling, scrapers, and scripts that kill repetitive work. If you need a data-extraction tool wired into your real systems, that's the work I do.
- Upwork: its_weli on Upwork
- Contact: hello@weli.build
MIT — use it, fork it, ship it.