Big Data
Unit 1
Extremely large and complex datasets that exceed the processing capability of traditional database tools. Characterized by volume, velocity, variety, veracity, and value.
5 Vs of Big Data
Unit 1
Volume (amount), Velocity (speed of generation), Variety (types), Veracity (quality/trustworthiness), and Value (usefulness) — the five defining characteristics of big data.
Structured Data
Unit 1
Data organized in a predefined format with rows and columns, easily stored in relational databases. Examples include spreadsheets, CSV files, and SQL tables.
Semi-structured Data
Unit 1
Data that does not reside in a formal relational database but has some organizational properties like tags or markers. Examples include JSON, XML, and HTML files.
Unstructured Data
Unit 1
Data that lacks a predefined format or structure. Examples include text documents, images, videos, audio files, and social media posts. Constitutes ~80% of all data.
Data Lake
Unit 1
A centralized storage repository that holds vast amounts of raw data in its native format until needed for analysis. Supports structured, semi-structured, and unstructured data.
Data Warehouse
Unit 1
A system used for reporting and data analysis that stores structured, processed data optimized for query performance rather than transaction processing.
Data Mining
Unit 1
The process of discovering patterns, anomalies, and correlations within large datasets to predict outcomes using statistical and mathematical techniques.
ETL
Unit 1
Extract, Transform, Load — a process that extracts data from sources, transforms it into a suitable format, and loads it into a target data warehouse or data lake.
HDFS
Unit 2
Hadoop Distributed File System — a distributed, scalable filesystem designed to store very large datasets across clusters of commodity machines with high throughput access.
MapReduce
Unit 2
A programming model for processing large datasets in parallel across a Hadoop cluster. It consists of a Map phase (filtering and sorting) and a Reduce phase (aggregation).
YARN
Unit 2
Yet Another Resource Negotiator — Hadoop's cluster resource management system that schedules jobs and manages cluster resources, enabling multiple processing engines.
Hadoop
Unit 2
An open-source framework for distributed storage and processing of large datasets using the MapReduce programming model and HDFS for storage.
NameNode
Unit 2
The master node in HDFS that manages the filesystem namespace and regulates access to files. It maintains the directory tree of all files in HDFS.
DataNode
Unit 2
The worker node in HDFS that stores actual data blocks on local disks. DataNodes serve read/write requests and perform block replication as instructed by the NameNode.
HDFS Block
Unit 2
The basic unit of data storage in HDFS, typically 128 MB or 256 MB in size. Files are split into blocks and distributed across DataNodes.
Replication
Unit 2
The process of copying HDFS blocks across multiple DataNodes (default factor: 3) to ensure fault tolerance and data reliability.
InputSplit
Unit 2
A logical representation of the data to be processed by a single Mapper in MapReduce. It defines the boundary of data that each map task processes.
Apache Spark
Unit 3
A fast, general-purpose, distributed data processing engine that supports in-memory processing for speeds up to 100x faster than MapReduce for certain workloads.
RDD (Resilient Distributed Dataset)
Unit 3
The fundamental data structure in Spark — an immutable, partitioned collection of elements that can be operated on in parallel with automatic fault recovery.
DataFrame
Unit 3
A distributed collection of data organized into named columns, similar to a table in a relational database. Provides optimized execution through the Catalyst optimizer.
Dataset
Unit 3
A strongly-typed, immutable collection of data with functional transformations. Combines the benefits of RDDs (type safety, lambda functions) with DataFrame optimizations.
Spark SQL
Unit 3
A Spark module for structured data processing that provides a programming interface to work with structured data using SQL queries and the DataFrame API.
Spark Streaming / Structured Streaming
Unit 3
Extensions of core Spark for processing real-time data streams. Structured Streaming treats a live data stream as a continuously appending unbounded table.
MLlib
Unit 3
Spark's scalable machine learning library providing algorithms for classification, regression, clustering, collaborative filtering, and feature engineering.
SparkContext / SparkSession
Unit 3
The entry point to Spark functionality. SparkSession (in Spark 2+) unifies SparkContext, SQLContext, and HiveContext into a single entry point for interacting with Spark.
Lazy Evaluation
Unit 3
Spark does not execute transformations immediately but builds a DAG (Directed Acyclic Graph) of operations. Execution is triggered only when an action is called.
DStream (Discretized Stream)
Unit 3
The basic abstraction in Spark Streaming representing a continuous stream of data as a sequence of RDDs, each representing data from a specific time interval.
NoSQL
Unit 4
A class of database management systems that do not use the traditional relational model. Designed for distributed data stores, horizontal scaling, and flexible schemas.
MongoDB
Unit 4
A document-oriented NoSQL database that stores data in JSON-like BSON documents. Supports rich queries, indexing, aggregation, and horizontal scaling via sharding.
Apache Cassandra
Unit 4
A distributed, wide-column store NoSQL database designed for high availability and linear scalability with no single point of failure. Uses CQL (Cassandra Query Language).
Apache HBase
Unit 4
A distributed, column-oriented NoSQL database built on top of HDFS. Provides real-time read/write access to large datasets and is modeled after Google's Bigtable.
CAP Theorem
Unit 4
A distributed system can provide at most two out of three guarantees: Consistency (all nodes see the same data), Availability (every request gets a response), and Partition tolerance (system works despite network failures).
Sharding
Unit 4
A database partitioning technique that distributes data across multiple machines (shards) to achieve horizontal scalability and improved performance for large datasets.
Replication Strategy
Unit 4
The method by which data copies are distributed across nodes in a distributed database. Strategies include SimpleStrategy and NetworkTopologyStrategy in Cassandra.
Key-Value Store
Unit 4
A simple NoSQL data model where data is stored as a collection of key-value pairs. Examples include Redis, DynamoDB, and Riak.
Eventual Consistency
Unit 4
A consistency model where, if no new updates are made, all replicas will eventually converge to the same state. Prioritizes availability over strong consistency.
Descriptive Analytics
Unit 5
Analysis of historical data to understand what has happened in the past. Uses summary statistics, charts, and reports to describe patterns and trends.
Diagnostic Analytics
Unit 5
Analysis that investigates why something happened. Uses drill-down, data discovery, and correlations to identify root causes of trends or anomalies.
Predictive Analytics
Unit 5
Analysis that uses statistical models and machine learning techniques to forecast what is likely to happen in the future based on historical data patterns.
Prescriptive Analytics
Unit 5
Analysis that recommends actions to achieve desired outcomes. Uses optimization and simulation algorithms to suggest the best course of action.
Data Visualization
Unit 5
The graphical representation of data and information using charts, graphs, maps, and dashboards to communicate insights clearly and effectively.
Key Performance Indicator (KPI)
Unit 5
A quantifiable measure used to evaluate the success of an organization, employee, or process in meeting objectives. KPIs align with strategic goals.
A/B Testing
Unit 5
A controlled experiment comparing two versions (A and B) to determine which performs better on a given metric. Commonly used in web optimization and marketing.
Regression Analysis
Unit 5
A statistical method for estimating relationships between a dependent variable and one or more independent variables. Used for prediction and trend analysis.