Uber uses Apache Spark Streaming to process millions of GPS pings, ride requests, and traffic signals every second. This data feeds into a dynamic "surge pricing" algorithm that adjusts fares in real-time based on supply and demand across every city zone.
Key Facts:
Processes 1+ million GPS events per second globally
Pricing decisions made in under 200 milliseconds
Spark Streaming with micro-batches every 2 seconds
Integrates traffic, weather, and event data simultaneously
Reduced average rider wait time by 30% in pilot cities
Focus Question:
Why is Spark Streaming preferred over batch processing for Uber's pricing system?
Tip: Spark Streaming uses micro-batching — processing small chunks of data every few seconds — striking a balance between true real-time (Storm/Flink) and traditional batch processing.
E-Commerce
Amazon Product Recommendations
Amazon's recommendation engine runs on Apache Spark, using collaborative filtering and matrix factorization on billions of user-item interactions. The system analyzes browsing history, purchase patterns, and product similarity to generate personalized "Customers who bought this also bought" suggestions in near real-time.
Key Facts:
Analyzes 350+ million active customer accounts
Recommendations drive 35% of total sales revenue
Processes 1+ billion data points daily on Spark
ALS (Alternating Least Squares) algorithm for collaborative filtering
Model retrained nightly on updated interaction matrix
Focus Question:
Which Spark component is best suited for the iterative matrix factorization computation?
Tip: Spark MLlib provides scalable implementations of algorithms like ALS that benefit from Spark's in-memory processing, making iterative machine learning on massive datasets efficient.
Social Media
Twitter Trending Topics
Twitter (now X) uses Spark to detect trending topics in real-time by analyzing tweet volume, retweet velocity, hashtag co-occurrence, and geolocation data. The system identifies sudden spikes in conversation topics across millions of tweets per minute, surfacing trends with a lag of under 5 minutes.
Key Facts:
Processes 500+ million tweets per day
Detects trends within 5 minutes of a topic spike
Spark Streaming with Kafka for event ingestion
Uses sliding windows of 15-minute intervals for trend scoring
Filters spam and bots using NLP-based veracity checks
Focus Question:
What role does the sliding window play in Twitter's trend detection pipeline?
Tip: Spark Streaming's window operations (window, countByWindow, reduceByKeyAndWindow) allow you to aggregate data over rolling time intervals — perfect for detecting patterns like trending topics.