{
"cells": [
{
"cell_type": "markdown",
"id": "329d05ab",
"metadata": {},
"source": [
"# Lesson 15: Data wrangling demonstration\n",
"\n",
"This notebook demonstrates key data wrangling techniques for handling missing data and outliers.\n",
"\n",
"**1. Handling missing data**\n",
"\n",
"- **Mean imputation**: Replace missing values with the column mean\n",
"- **Median imputation**: Replace missing values with the column median\n",
"- **KNN imputation**: Use K-Nearest Neighbors to estimate missing values\n",
"- **Iterative imputation**: Model each feature as a function of others (MICE algorithm)\n",
"\n",
"**2. Detecting outliers**\n",
"\n",
"- **IQR method**: Use interquartile range (Q3 - Q1) to identify outliers\n",
"- **Z-score method**: Identify values more than 3 standard deviations from the mean\n",
"- **Isolation Forest**: Machine learning algorithm that isolates anomalies\n",
"- **Local Outlier Factor (LOF)**: Density-based method comparing local density to neighbors\n",
"\n",
"**3. Handling outliers**\n",
"\n",
"- **Removal**: Remove outlier observations from the dataset\n",
"- **Capping (Winsorization)**: Replace outliers with boundary values (5th and 95th percentiles)\n",
"- **Binning (Discretization)**: Convert continuous values into discrete bins\n",
"- **Log transformation**: Apply logarithmic transformation to compress value range"
]
},
{
"cell_type": "markdown",
"id": "d0dd917a",
"metadata": {},
"source": [
"## Setup\n",
"\n",
"### Imports"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "e85ff7ed",
"metadata": {},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"from matplotlib.gridspec import GridSpec\n",
"from scipy import stats\n",
"from sklearn.ensemble import IsolationForest\n",
"from sklearn.experimental import enable_iterative_imputer\n",
"from sklearn.impute import KNNImputer, IterativeImputer\n",
"from sklearn.neighbors import LocalOutlierFactor\n",
"from sklearn.preprocessing import KBinsDiscretizer"
]
},
{
"cell_type": "markdown",
"id": "50d8e32f",
"metadata": {},
"source": [
"### Load datasets"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "f4509aa8",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" MedInc | \n",
" HouseAge | \n",
" AveRooms | \n",
" AveBedrms | \n",
" Population | \n",
" AveOccup | \n",
" Latitude | \n",
" Longitude | \n",
" MedHouseVal | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" 8.3252 | \n",
" 41.0 | \n",
" 6.984127 | \n",
" 1.023810 | \n",
" 322.0 | \n",
" 2.555556 | \n",
" 37.88 | \n",
" -122.23 | \n",
" 4.526 | \n",
"
\n",
" \n",
" | 1 | \n",
" 8.3014 | \n",
" 21.0 | \n",
" 6.238137 | \n",
" 0.971880 | \n",
" 2401.0 | \n",
" 2.109842 | \n",
" 37.86 | \n",
" -122.22 | \n",
" 3.585 | \n",
"
\n",
" \n",
" | 2 | \n",
" 7.2574 | \n",
" 52.0 | \n",
" 8.288136 | \n",
" 1.073446 | \n",
" 496.0 | \n",
" 2.802260 | \n",
" 37.85 | \n",
" NaN | \n",
" 3.521 | \n",
"
\n",
" \n",
" | 3 | \n",
" 5.6431 | \n",
" 52.0 | \n",
" 5.817352 | \n",
" 1.073059 | \n",
" 558.0 | \n",
" 2.547945 | \n",
" 37.85 | \n",
" -122.25 | \n",
" 3.413 | \n",
"
\n",
" \n",
" | 4 | \n",
" 3.8462 | \n",
" 52.0 | \n",
" 6.281853 | \n",
" NaN | \n",
" 565.0 | \n",
" 2.181467 | \n",
" 37.85 | \n",
" NaN | \n",
" 3.422 | \n",
"
\n",
" \n",
"
\n",
"