{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a92ddba5",
   "metadata": {},
   "source": [
    "# Lesson 22 activity: unsupervised learning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28c8b549",
   "metadata": {},
   "source": [
    "## Notebook setup\n",
    "\n",
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7248ac92",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0320dae3",
   "metadata": {},
   "source": [
    "### Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "77b65c19",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>total_rooms</th>\n",
       "      <th>total_bedrooms</th>\n",
       "      <th>population</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>median_house_value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-122.23</td>\n",
       "      <td>37.88</td>\n",
       "      <td>41.0</td>\n",
       "      <td>880.0</td>\n",
       "      <td>129.0</td>\n",
       "      <td>322.0</td>\n",
       "      <td>126.0</td>\n",
       "      <td>8.3252</td>\n",
       "      <td>452600.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-122.22</td>\n",
       "      <td>37.86</td>\n",
       "      <td>21.0</td>\n",
       "      <td>7099.0</td>\n",
       "      <td>1106.0</td>\n",
       "      <td>2401.0</td>\n",
       "      <td>1138.0</td>\n",
       "      <td>8.3014</td>\n",
       "      <td>358500.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-122.24</td>\n",
       "      <td>37.85</td>\n",
       "      <td>52.0</td>\n",
       "      <td>1467.0</td>\n",
       "      <td>190.0</td>\n",
       "      <td>496.0</td>\n",
       "      <td>177.0</td>\n",
       "      <td>7.2574</td>\n",
       "      <td>352100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-122.25</td>\n",
       "      <td>37.85</td>\n",
       "      <td>52.0</td>\n",
       "      <td>1274.0</td>\n",
       "      <td>235.0</td>\n",
       "      <td>558.0</td>\n",
       "      <td>219.0</td>\n",
       "      <td>5.6431</td>\n",
       "      <td>341300.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-122.25</td>\n",
       "      <td>37.85</td>\n",
       "      <td>52.0</td>\n",
       "      <td>1627.0</td>\n",
       "      <td>280.0</td>\n",
       "      <td>565.0</td>\n",
       "      <td>259.0</td>\n",
       "      <td>3.8462</td>\n",
       "      <td>342200.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   longitude  latitude  housing_median_age  total_rooms  total_bedrooms  \\\n",
       "0    -122.23     37.88                41.0        880.0           129.0   \n",
       "1    -122.22     37.86                21.0       7099.0          1106.0   \n",
       "2    -122.24     37.85                52.0       1467.0           190.0   \n",
       "3    -122.25     37.85                52.0       1274.0           235.0   \n",
       "4    -122.25     37.85                52.0       1627.0           280.0   \n",
       "\n",
       "   population  households  median_income  median_house_value  \n",
       "0       322.0       126.0         8.3252            452600.0  \n",
       "1      2401.0      1138.0         8.3014            358500.0  \n",
       "2       496.0       177.0         7.2574            352100.0  \n",
       "3       558.0       219.0         5.6431            341300.0  \n",
       "4       565.0       259.0         3.8462            342200.0  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('https://gperdrizet.github.io/FSA_devops/assets/data/unit3/housing.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de1e8cc9",
   "metadata": {},
   "source": [
    "## Task 1: prepare data for clustering\n",
    "\n",
    "Prepare the housing dataset for k-means clustering by completing the following steps:\n",
    "\n",
    "- Check for and handle any missing values in the dataset\n",
    "- Standardize the features using `StandardScaler` to ensure all variables are on the same scale"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ed7fac23",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4541c37d",
   "metadata": {},
   "source": [
    "## Task 2: cluster data with k-means\n",
    "\n",
    "Apply k-means clustering to the prepared data:\n",
    "\n",
    "- Use the elbow method to determine the optimal number of clusters (k)\n",
    "  - Test k values from 2 to 10\n",
    "  - Plot the within-cluster sum of squares (inertia) for each k value\n",
    "  - Choose an appropriate k value based on the elbow in the plot\n",
    "- Fit a `KMeans` model with the chosen k value using `random_state=315`\n",
    "- Add the cluster labels as a new column to the original dataframe\n",
    "- Create a visualization to show the distribution of data points across clusters\n",
    "\n",
    "**Hints:**\n",
    "- The `inertia_` attribute of a fitted KMeans model gives you the within-cluster sum of squares\n",
    "- A simple loop can help you test multiple k values and store their inertia values\n",
    "- Use `sklearn.manifold.TSNE` to collapse the features into two dimensions for plotting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1f061fe1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dc46951",
   "metadata": {},
   "source": [
    "## Task 3: train a supervised model to assign cluster membership\n",
    "\n",
    "Build a supervised learning model that can predict cluster assignments for new data:\n",
    "\n",
    "- Split the data into training and testing sets`\n",
    "- Use the scaled features from task 1 as the input and the cluster labels from task 2 as the label\n",
    "- Train a classification model (such as `RandomForestClassifier` or `LogisticRegression`) to predict cluster membership\n",
    "- Evaluate the model's performance on the test set using appropriate metrics (F1 score, precision, recall, confusion matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b1f4da2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}