- Case study · Data
- 6 technologies
ParseHub: 2 million records a day without getting blocked
Elvisupdated 5 min read
A platform that collects data from websites that have no API. Rate limits adjust to each source, data arrives in one format, and you can watch the work in a real-time monitor.
The challenge
The business needed data from other people's websites, on a regular schedule and in large volumes. Most of those sites have no proper API, and the data exists only on the pages, often loaded by scripts.
A one-off script couldn't handle this. It broke whenever the page layout changed, failed without warning and returned data in a different shape from every source. The business needed a system that reliably collects large volumes every day and delivers clean results you can work with right away.
Collecting by hand wasn't an option either. A person can't get through that many pages every day, and copying by hand always leads to gaps and typos. On top of that, the data has to be fresh, and a manual export goes stale while it's being made.
The solution
I built the platform in Python. Pages that render in the browser are handled by Puppeteer: it opens the page like a regular browser and sees the data that scripts load. Control and data delivery go through an API built on FastAPI.
Collected data is normalized and stored in PostgreSQL, and Redis holds the task queue. The whole system runs in Docker, so it's easy to deploy and scale. More than two million records a day go through the platform.
From the start I assumed sources would change and new ones would be added. That's why collection, normalization, storage and delivery are separate, and changing one part doesn't break the rest.
How the system works
Scraping websites that have no API
Many sites only show their data in the browser, and a plain request to the page returns an empty template. Puppeteer solves this: it waits until the page has fully loaded and only then grabs the fields you need.
Each source has its own collection rules. If a site changes its layout, you update the rules for that one source while the rest keep running.
Smart rate limiting for each source
Every site can handle its own request rate. The platform adjusts limits to each source so collection runs steadily and doesn't overload the site the data comes from. A steady pace helps the platform too: data comes in evenly instead of in bursts.
Data normalization: one format for every source
Each site has its own way of writing dates, prices and names. Before saving, the platform converts the data to one format, so records from different sources sit in PostgreSQL in the same structure.
The client gets a ready spreadsheet or API instead of a pile of raw exports that still need cleaning.
Real-time scraping monitor
The monitor shows which source is being collected right now, how many records have come in and where errors happened. If a source stops returning data, you see it right away, not a week later from an empty report. Platform uptime is 99.9%.
FastAPI data API running in Docker
Collection control and data delivery go through an API built on FastAPI. The client's systems connect to it and pull finished records directly, without manual exports or emailing files around. The response format is the same for every source.
Each part of the platform runs in its own Docker container. You can add a new source or more capacity without stopping what's already running.
Results
- records/day
- 2M+
- uptime
- 99.9%
- 2M+ records a day
- 99.9% uptime
- Data arrives in one format
Tech stack
- Python
- FastAPI
- Puppeteer
- Redis
- PostgreSQL
- Docker
Who it is for
Companies that need data from other websites as a clean spreadsheet or API, on a regular schedule and in large volumes.
Project FAQ
What format will I get the data in?
One consistent format, as a spreadsheet or through an API. Data is normalized before it's saved, so records from different sites can be compared and loaded into your systems right away.
Can you collect data from the sites I need?
Usually, yes. Each source gets its own collection rules and limits. Send me a list of sites and the fields you need on Telegram, and I'll tell you what can realistically be collected.
What is the scraper built with?
Python: FastAPI for the API, Puppeteer for pages that render in the browser, PostgreSQL for storage and Redis for the task queue. Everything runs in Docker.
How long does it take to start collecting data?
It depends on how many sources there are, how they're built and what output format you need. Message me on Telegram, we'll go through the sources and estimate the timeline.
Ready to automate your business?
A 30-minute call: we go through your processes, pick what to automate first and estimate the timeline.
- CRM
- Traffic
- Bots
- AI
- Analytics
- Integrations
- Websites
Need something similar
- DataAnalytics PlatformAn analytics platform with live dashboards, forecasts and custom reports. Revenue data sits in one place, so nobody has to build a report by hand before making a decision.
- Data · built-in analyticsLink Bio + AnalyticsA link-in-bio page for a model with links, tips and built-in analytics. The fan comes in and tips right on the page, while the model sees revenue and traffic sources in her own admin panel.
- CRM · 29+ modelsOnlyWorkPlaceA CRM for an OnlyFans agency with a team of chatters working in shifts. Chatters see their shift and earnings, managers watch every shift as it happens, and the owner sees revenue by model and ad ROI on one screen.
