System Design Case Studies · Part 6 of 7
System Design - Web Crawler
Contents
In this wiki, we will explore an approach to designing a Web-crawling service.
Requirements
- What is the purpose of the crawler here? Search engine indexing, data mining, or for some other purpose? => Search engine indexing
- Type of content that needs to be parsed: HTML/ Text/ PDF/ Image/ Video => HTML only
- How long do we need to store the parsed content? => 5 years
- How do we handle a web page that is edited? => Web-page is parsed only once
- Scale of web-crawling? => 1 billion pages per month
- Robustness: Need to handle the edge cases while handling web pages, including non-responsiveness, web server crashing, bad HTML, etc.
- Politeness: Shouldn’t make too many requests to a website within a short span of time. The service can get marked as a DoS attacker.
- Extensibility: Flexible enough to support parsing new content types with minimal changes
Back of the envelope estimation:
- 1 billion web pages to be downloaded each month
- QPS => 1 billion pages / 30 days / 24 hours / 3600 seconds => ~400 pages/second
- Peak QPS = 2 * QPS => 800
- Assume the average web page size is 500k
- Storage requirement = 1 billion page/month * 500k => 500TB/month
- Total Storage requirement = 500 TB * 12 months * 5 years => 30 PB
Architecture
Architecture: web crawler components
Architecture: web crawler hld
Spider Trap: Causes the crawler to loop in an infinite loop. Need to handle such edge cases.
Future Study:
- How to know a webpage is edited? Any optimization regarding this?
- How are bloom filters and hash values used? Check examples