Hadoop is an open-source software framework designed for distributed storage and processing of large datasets across clusters of computers. It was created by Doug Cutting and Mike Cafarella in 2006 and is named after a toy elephant.
Hadoop enables the processing of massive datasets in parallel across clusters of commodity hardware, making it cost-effective for organizations dealing with petabytes of data. It provides fault tolerance through data replication and automatic recovery of failed tasks.
The framework follows a simple principle: move computation to the data rather than moving data to the computation. This approach minimizes network overhead and improves processing efficiency significantly.
Key Point: Hadoop is built on four core modules: HDFS (storage), MapReduce (processing), YARN (resource management), and Hadoop Common (utilities). It can scale from a single server to thousands of machines, each offering local computation and storage.
1
Hadoop Architecture Overview
graph TD
A[Hadoop Framework] --> B[HDFS]
A --> C[MapReduce]
A --> D[YARN]
A --> E[Hadoop Common]
B --> B1[NameNode]
B --> B2[DataNodes]
B1 --> B3[Master Metadata]
B2 --> B4[Block Storage]
C --> C1[Map Phase]
C --> C2[Reduce Phase]
C1 --> C3[Input Splitting]
C2 --> C4[Output Aggregation]
D --> D1[ResourceManager]
D --> D2[NodeManagers]
D1 --> D3[Scheduler]
D2 --> D4[Container Execution]
E --> E1[Libraries & Utilities]
E --> E2[Configuration & CLI]
classDef decision fill:#6366f1,stroke:#818cf8,color:#fff,font-size:18px,font-weight:bold
classDef process fill:#0ea5e9,stroke:#38bdf8,color:#fff,font-size:18px,font-weight:bold
classDef success fill:#22c55e,stroke:#4ade80,color:#fff,font-size:18px,font-weight:bold
classDef warning fill:#f59e0b,stroke:#fbbf24,color:#1e1b4b,font-size:18px,font-weight:bold
classDef info fill:#8b5cf6,stroke:#a78bfa,color:#fff,font-size:18px,font-weight:bold
class A decision
class B,C,D,E process
class B1,B2,C1,C2,D1,D2,E1,E2 success
class B3,B4,C3,C4,D3,D4,E2 info
Hadoop's architecture is built on four core modules that work together to provide scalable, reliable, and fault-tolerant storage and processing capabilities for big data workloads.
2. Hadoop Distributed File System (HDFS)
HDFS is the primary storage system of Hadoop, designed to store very large files reliably across clusters of machines. It provides high-throughput access to application data and is optimized for batch processing workloads.
HDFS follows a master-slave architecture where the NameNode (master) manages the file system namespace and regulates access to files, while DataNodes (slaves) store actual data blocks. Files are split into 128MB or 256MB blocks that are replicated across multiple DataNodes for fault tolerance.
HDFS is designed to run on commodity hardware, making it cost-effective for large-scale storage. It provides stream access to file system data, making it suitable for batch processing rather than random access patterns.
Key Point: HDFS replicates each data block 3 times by default across different DataNodes. This replication ensures data availability even if individual nodes fail. The NameNode keeps metadata in memory for fast access, supporting millions of files and blocks.
2
HDFS Architecture
graph TD
A[Client Application] -->|Write Request| B[NameNode]
A -->|Read/Write| C[DataNodes]
B -->|Metadata| D[File System Namespace]
B -->|Block Mapping| E[Block Reports]
C --> F[DataNode 1]
C --> G[DataNode 2]
C --> H[DataNode 3]
F --> F1[Block A Replica]
F --> F2[Block B Replica]
G --> G1[Block A Replica]
G --> G2[Block C Replica]
H --> H1[Block B Replica]
H --> H2[Block C Replica]
B --> I[Edit Logs & Checkpoints]
D --> D1[File → Block Mapping]
E --> E1[Heartbeat Signals]
classDef decision fill:#6366f1,stroke:#818cf8,color:#fff,font-size:18px,font-weight:bold
classDef process fill:#0ea5e9,stroke:#38bdf8,color:#fff,font-size:18px,font-weight:bold
classDef success fill:#22c55e,stroke:#4ade80,color:#fff,font-size:18px,font-weight:bold
classDef warning fill:#f59e0b,stroke:#fbbf24,color:#1e1b4b,font-size:18px,font-weight:bold
classDef info fill:#8b5cf6,stroke:#a78bfa,color:#fff,font-size:18px,font-weight:bold
class A decision
class B process
class C,F,G,H success
class D,E,I warning
class D1,E1,F1,F2,G1,G2,H1,H2 info
HDFS splits files into blocks and distributes them across DataNodes with replication. The NameNode maintains metadata and handles client requests, while DataNodes store the actual data blocks and report their status.
3. MapReduce
MapReduce is a programming model and processing engine for distributed processing of large datasets. It breaks down complex problems into two phases: Map (filter and sort) and Reduce (aggregate results).
In the Map phase, input data is split into chunks and processed in parallel across the cluster. Each mapper processes a chunk and produces key-value pairs as intermediate output. The framework then shuffles and sorts these pairs by key before passing them to reducers.
In the Reduce phase, all values associated with the same key are grouped together and processed by a single reducer. This enables aggregation, summarization, and other computations on the grouped data. The final output is stored in HDFS.
Key Point: MapReduce automatically handles parallelization, fault tolerance, and data distribution. Programmers only need to define the Map and Reduce functions. This abstraction allows developers to write distributed applications without worrying about network communication, data transfer, or fault recovery.
4. YARN (Yet Another Resource Negotiator)
YARN is Hadoop's resource management layer introduced in Hadoop 2.0. It separates resource management from data processing, enabling multiple processing frameworks to share the same cluster efficiently.
YARN consists of two main components: the ResourceManager (global resource broker) and NodeManagers (per-node agents). The ResourceManager allocates resources to applications, while NodeManagers monitor and manage containers on individual nodes.
With YARN, Hadoop supports not just MapReduce but also other frameworks like Apache Spark, Apache Flink, and Apache Tez. This flexibility makes the cluster more versatile and improves resource utilization.
Key Point: YARN enables multi-tenancy on Hadoop clusters. It supports FIFO, Capacity, and Fair scheduling policies to allocate resources based on organizational needs. This allows different teams and applications to share cluster resources efficiently.
5. Hadoop Ecosystem Tools
The Hadoop ecosystem includes various tools that extend its capabilities for different use cases:
Apache Hive: SQL-like interface for querying data stored in HDFS. Hive translates queries to MapReduce jobs, making it easier for analysts familiar with SQL.
Apache HBase: NoSQL database built on HDFS for real-time read/write access to large datasets. Suitable for random access patterns that HDFS doesn't support.
Apache Pig: High-level scripting language (Pig Latin) for data transformation and analysis. Simplifies complex MapReduce operations.
Apache Sqoop: Tool for importing and exporting data between Hadoop and relational databases. Enables efficient bulk data transfer.
Apache Flume: Distributed service for collecting, aggregating, and moving large amounts of log data into HDFS in real-time.
Apache Oozie: Workflow scheduler for managing Hadoop jobs. Defines complex workflows with dependencies and scheduling policies.
Apache Kafka: Distributed streaming platform for building real-time data pipelines. Integrates with Hadoop for stream processing.
Key Point: Each ecosystem tool addresses specific needs - from data ingestion (Sqoop, Flume) to processing (Hive, Pig) to storage (HBase) and workflow management (Oozie). Together, they form a complete big data platform for various business requirements.
3
Hadoop Ecosystem Tools
graph TD
A[Hadoop Ecosystem] --> B[Storage Layer]
A --> C[Processing Layer]
A --> D[Data Ingestion]
A --> E[Management & Scheduling]
B --> B1[HDFS]
B --> B2[HBase]
B --> B3[Amazon S3]
C --> C1[MapReduce]
C --> C2[Apache Spark]
C --> C3[Apache Hive]
C --> C4[Apache Pig]
D --> D1[Apache Sqoop]
D --> D2[Apache Flume]
D --> D3[Apache Kafka]
E --> E1[Apache Oozie]
E --> E2[Apache ZooKeeper]
E --> E3[Apache Ambari]
B1 --> F[Distributed Storage]
C2 --> G[In-Memory Processing]
D3 --> H[Real-time Streaming]
E3 --> I[Cluster Management]
classDef decision fill:#6366f1,stroke:#818cf8,color:#fff,font-size:18px,font-weight:bold
classDef process fill:#0ea5e9,stroke:#38bdf8,color:#fff,font-size:18px,font-weight:bold
classDef success fill:#22c55e,stroke:#4ade80,color:#fff,font-size:18px,font-weight:bold
classDef warning fill:#f59e0b,stroke:#fbbf24,color:#1e1b4b,font-size:18px,font-weight:bold
classDef info fill:#8b5cf6,stroke:#a78bfa,color:#fff,font-size:18px,font-weight:bold
class A decision
class B,C,D,E process
class B1,B2,B3,C1,C2,C3,C4,D1,D2,D3,E1,E2,E3 success
class F,G,H,I info
The Hadoop ecosystem provides a comprehensive suite of tools for every stage of the big data pipeline - from data ingestion and storage to processing, analysis, and cluster management.