Lab Manual

Load, explore, and analyze large datasets using Python libraries. Perform data cleaning, handle missing values, and generate summary statistics for big data analytics.

1. Setup: Install Python 3.x, Pandas, NumPy, and Matplotlib
2. Load Data: Use pd.read_csv() to load a large dataset (e.g., NYC Taxi Trips)
3. Explore: Run df.head(), df.info(), df.describe()
4. Clean: Handle missing values with dropna() or fillna()
5. Transform: Convert data types, create new columns, filter rows
6. Analyze: Group by columns, compute aggregates, find correlations
7. Visualize: Plot histograms, box plots, and scatter plots
8. Report: Summarize findings and export cleaned data to CSV

Perform Hadoop Distributed File System operations and write MapReduce programs to process large datasets across a cluster.

1. Setup: Install Hadoop in pseudo-distributed mode and configure HDFS
2. HDFS Commands: Run hdfs dfs -mkdir /input and -put files to HDFS
3. Verify: Use hdfs dfs -ls /input to list uploaded files
4. MapReduce (WordCount): Write Mapper class to tokenize input lines
5. Reducer: Write Reducer class to aggregate word counts
6. Driver: Configure Job with input/output paths and jar file
7. Execute: Run hadoop jar command to submit the job
8. Output: Read results from HDFS using hdfs dfs -cat /output/part-r-00000

Create and manipulate Resilient Distributed Datasets (RDDs) in Spark. Perform transformations and actions for distributed data processing.

1. Setup: Install Apache Spark and configure PySpark environment
2. Create RDD: Use sc.textFile() or sc.parallelize()
3. Transformations: Apply map(), filter(), flatMap()
4. Actions: Use collect(), count(), take(), reduce()
5. Pair RDDs: Create key-value pairs and use reduceByKey(), groupByKey()
6. Persistence: Cache RDDs with cache() or persist()
7. WordCount: Implement word count using RDD transformations
8. Compare: Measure execution time vs MapReduce for same dataset

Use Spark SQL and DataFrames for structured data processing. Write SQL queries on large datasets and perform complex analytics.

1. Setup: Create SparkSession with SparkSession.builder
2. Load Data: Read JSON/CSV into DataFrame using spark.read
3. Schema: Define schema using StructType or infer from data
4. Operations: Use select(), filter(), groupBy(), agg()
5. SQL: Create temp view with createOrReplaceTempView() and run SQL
6. Join: Join multiple DataFrames using join()
7. Window Functions: Use row_number(), rank(), lag()
8. Output: Write results to Parquet format using write.parquet()

Perform Create, Read, Update, and Delete operations in MongoDB. Work with document-based storage for unstructured big data.

1. Setup: Install MongoDB and start the mongod service
2. Connect: Use pymongo.MongoClient() to connect to server
3. Create: Insert single/many documents with insert_one() / insert_many()
4. Read: Query with find(), find_one(), and query operators ($gt, $in, $regex)
5. Update: Use update_one() and update_many() with $set, $push
6. Delete: Remove documents with delete_one() / delete_many()
7. Indexing: Create indexes with create_index() for faster queries
8. Aggregation: Use aggregation pipeline with $match, $group, $sort

Design and implement data models in Apache Cassandra. Perform CRUD operations and understand partitioning strategies for distributed storage.

1. Setup: Install Apache Cassandra and start the service
2. Connect: Use cqlsh shell or Python cassandra-driver
3. Keyspace: Create keyspace with CREATE KEYSPACE (SimpleStrategy)
4. Table: Design table with primary key (partition key + clustering columns)
5. Insert: Add rows using INSERT INTO with proper partitioning
6. Query: Read data with SELECT (must include partition key)
7. Update/Delete: Use UPDATE and DELETE with partition key
8. Modeling: Create denormalized tables for different query patterns

Create interactive and static visualizations for big data analytics. Generate charts, graphs, and dashboards to communicate insights.

1. Setup: Install Matplotlib, Seaborn, and Plotly
2. Basic Plots: Create line plots, bar charts, and pie charts with plt.plot()
3. Subplots: Use plt.subplots() for multi-panel figures
4. Styling: Apply themes with seaborn.set_style() and custom colors
5. Distributions: Plot histograms with plt.hist() and KDE with seaborn
6. Heatmaps: Create correlation heatmaps using sns.heatmap()
7. Interactive: Build interactive plots with Plotly (plotly.express)
8. Export: Save figures as PNG/SVG with plt.savefig()

Implement K-Means clustering algorithm using Spark MLlib for large-scale unsupervised machine learning on distributed datasets.

1. Setup: Configure Spark with MLlib dependencies
2. Load Data: Read dataset into DataFrame and assemble feature vector
3. Preprocess: Use VectorAssembler and StandardScaler
4. Model: Create KMeans estimator with k=3 and fit on data
5. Evaluate: Compute Within Set Sum of Squared Errors (WSSSE)
6. Optimize: Try different k values and plot elbow curve
7. Visualize: Plot cluster assignments against original data
8. Interpret: Analyze cluster centers and label assignments

Build a complete pipeline: data ingestion, processing, storage, and visualization. Integrate Hadoop, Spark, NoSQL, and analytics tools.

1. Ingest: Upload raw data to HDFS from multiple sources (CSV, JSON, logs)
2. Process: Use Spark to clean, transform, and aggregate the data
3. Store: Write processed results to MongoDB and/or Cassandra
4. Index: Create appropriate indexes in NoSQL databases for fast queries
5. Analyze: Run Spark SQL queries and MLlib models on processed data
6. Visualize: Generate dashboards with Matplotlib/Plotly from stored data
7. Monitor: Track pipeline execution with Spark UI and logs
8. Document: Record metrics, performance benchmarks, and insights