Data Preprocessing & Feature Normalization with Pandas

A practical implementation of fundamental data cleaning and preprocessing techniques in Python using Pandas. This repository covers missing data imputation, duplicate record elimination, and feature scaling via Min-Max normalization.

---

Overview

In machine learning and statistical data analysis, raw datasets are rarely clean. They often contain missing entries, redundant records, and unscaled numerical features. This project demonstrates how to address these common issues systematically.

Core Objectives

1. Handling Missing Values (Imputation): Filling numerical missing values with the column mean and categorical null values with a constant placeholder ('Unknown').
2. Deduplication: Detecting and dropping redundant rows to prevent data leakage and bias.
3. Feature Scaling (Min-Max Normalization): Transforming numerical features into a standard range $[0, 1]$ to ensure uniform weight distribution across machine learning algorithms.

---

Mathematical Formulation

Min-Max Normalization

Min-Max Scaling shifts and rescales values so that they bound strictly within $[0, 1]$:

$$x_{\text{norm}} = \frac{x - x_{\min}}{x_{\max} - x_{\min}}$$

Where:

• $x$ is the original feature value.
• $x_{\min}$ is the minimum value in the feature column.
• $x_{\max}$ is the maximum value in the feature column.
• $x_{\text{norm}}$ is the rescaled value between $0.0$ and $1.0$.

Standard Scaling (Z-Score Standardization)

(Included in the modular pipeline)

$$z = \frac{x - \mu}{\sigma}$$

Where:

• $\mu$ is the column mean.
• $\sigma$ is the column standard deviation.

---

Repository Structure

pandas-data-preprocessing/
├── data/
│   ├── students_data.csv          # Sample dataset with duplicates and missing values
│   └── students_cleaned.csv       # Cleaned and normalized output dataset
├── data_preprocessing.py          # Direct assignment script with formatted output
├── data_preprocessing_pipeline.py # Production-ready modular preprocessing class
├── data_preprocessing.ipynb       # Interactive Jupyter Notebook walkthrough
├── requirements.txt               # Dependencies
├── .gitignore                     # Git ignore file for Python & Notebooks
└── README.md                      # Project documentation

---

Getting Started

1. Prerequisites

Ensure you have Python 3.9+ installed on your system.

2. Installation

Clone the repository and install the dependencies:

git clone https://github.com/ombichave999/pandas-data-preprocessing.git
cd pandas-data-preprocessing
pip install -r requirements.txt

---

Usage

Run the Basic Script

Executes the step-by-step cleaning on the sample DataFrame:

python data_preprocessing.py

Run the Reusable Pipeline

Loads raw CSV data, executes the preprocessing pipeline, and exports the clean dataset:

python data_preprocessing_pipeline.py

Run the Jupyter Notebook

For interactive experimentation or academic presentations:

jupyter notebook data_preprocessing.ipynb

---

Example Walkthrough & Output

1. Raw Data Input

Name	Age	Marks	Issue
Amit	22.0	85.0	Valid
Riya	24.0	90.0	Valid
Amit	22.0	85.0	Duplicate
Neha	NaN	78.0	Missing Age
NaN	26.0	NaN	Missing Name & Marks

---

2. Processed & Normalized Output

==================================================
 Final Processed Data
==================================================
   Name  Age  Marks  Age_Normalized  Marks_Normalized
   Amit 22.0   85.0           0.000          0.583333
   Riya 24.0   90.0           0.500          1.000000
   Neha 23.5   78.0           0.375          0.000000
Unknown 26.0   84.5           1.000          0.541667

---

Best Practices & Engineering Notes

Tip

Deduplication Before Imputation: When calculating statistical properties (e.g. mean, median) for imputation, duplicate entries artificially skew the calculations. In production pipelines, deduplicate the data first so that summary statistics accurately represent the underlying distribution.

---

License

MIT License. Free to use for educational and research purposes.