Design
Web Crawler
Design a scalable web crawler that discovers, fetches, and indexes web pages across billions of URLs.
This is a Low-Level Design (LLD) problem. New to LLD vs HLD? Start here.
01 - Why interviewers ask this
It tests distributed BFS/DFS traversal plus politeness constraints — interviewers check if you'd avoid hammering one domain while still crawling billions of pages efficiently.
02 - Where this system exists in the real world
You interact with this design every day.
Search engine web crawlers (Googlebot)
SEO auditing and site-monitoring tools
Price-comparison and data-aggregation crawlers
03 - What you'll master
Solve this once. Know it forever.
Design distributed URL frontier management
Respect politeness/rate limits per domain
Deduplicate URLs and detect crawl traps
04 - What you'll design
7 functional · 4 non-functional requirements.
Start from seed URLs and discover new URLs via link extraction
6 more requirements inside
Sign in to unlock the full brief and start designing
05 - Companies that ask this
You may face this exact question in your next interview.
Ready?
Open the canvas.
Design it yourself.
See all requirements, use staged hints, run your code, and compare with community discussions - all on the same canvas.
More in Infrastructure