Facebook operates one of the largest Hadoop clusters in the world to store and process petabytes of user-generated data including photos, messages, and activity logs. The Hadoop Distributed File System (HDFS) enables Facebook to scale storage across thousands of commodity servers while MapReduce and later Hive power batch analytics for business intelligence and ad targeting.
Key Facts:
Stores over 300 petabytes of data across 2 million+ CPU cores
Runs the world's largest Hadoop deployment with 11,000+ machines
Processes 500+ terabytes of new data daily
Pioneered Hive – a data warehouse abstraction layer on top of Hadoop
Reduced query time from days to hours for ad-hoc analytics
Focus Question:
Why is Hadoop's HDFS better suited for Facebook's data than a traditional relational database?
Tip: HDFS is designed for write-once-read-many workloads and scales linearly by adding commodity machines – ideal for Facebook's petabyte-scale data lake.
Technology
Yahoo! Search Analytics
Yahoo! was one of the earliest and largest adopters of Apache Hadoop, using it to power search indexing, spam detection, and advertising analytics. Hadoop enabled Yahoo! to process massive web crawl datasets and ranking signals across clusters spanning thousands of nodes, significantly improving search relevance and operational efficiency.
Key Facts:
Operated a 42,000+ node Hadoop cluster – one of the largest at the time
Processed 100+ petabytes of web crawl and user interaction data
Used Hadoop for search ranking, ad targeting, and anti-spam filtering
Contributed key Hadoop patches including Fair Scheduler and Security
Reduced data processing costs by 50% compared to legacy systems
Focus Question:
What Hadoop feature enabled Yahoo! to distribute computation across thousands of machines?
Tip: MapReduce splits a job into Map and Reduce phases, allowing automatic parallelization of computation across every node in the Hadoop cluster.
Social Media
LinkedIn People You May Know
LinkedIn's "People You May Know" (PYMK) feature uses Hadoop to analyze massive social graphs and recommend relevant connections. The system processes billions of profile relationships, employment histories, and shared attributes to surface personalized connection suggestions that drive user engagement and network growth.
Key Facts:
Processes a social graph with 900+ million member profiles
Generates 40% of all new connections through PYMK recommendations
Runs batch MapReduce jobs to compute graph traversals on Hadoop
Incorporates mutual connections, company affiliations, and skill similarity
Scaled from single-machine processing to 4,000+ Hadoop nodes
Focus Question:
Which Hadoop component stores and manages LinkedIn's massive social graph data?
Tip: HDFS provides the distributed storage foundation that enables LinkedIn to store and access its massive social graph across thousands of machines, while MapReduce performs the graph computation.