{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6a3fc02e",
   "metadata": {},
   "source": [
    "# Lesson 26: multilayer perceptron activity\n",
    "\n",
    "## Notebook set up\n",
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4186739",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Third party imports\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import f1_score, roc_auc_score, log_loss, ConfusionMatrixDisplay\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn.preprocessing import StandardScaler, OrdinalEncoder, OneHotEncoder, MinMaxScaler"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ced7ed87",
   "metadata": {},
   "source": [
    "## 1. Data preparation\n",
    "\n",
    "### 1.1. Load diabetes dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c04a2b33",
   "metadata": {},
   "outputs": [],
   "source": [
    "diabetes_df = pd.read_csv('https://gperdrizet.github.io/FSA_devops/assets/data/unit3/diabetes_prediction_train.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a94313df",
   "metadata": {},
   "outputs": [],
   "source": [
    "diabetes_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a397236",
   "metadata": {},
   "outputs": [],
   "source": [
    "diabetes_df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "735e8b6c",
   "metadata": {},
   "source": [
    "Define the label and feature columns below. The label is `diabetes` (binary: 0 or 1). Using separate lists for numerical, nominal and ordinal features makes preprocessing easier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0e50bde",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the label\n",
    "label = # YOUR CODE HERE\n",
    "\n",
    "# Define numerical, ordinal and nominal features\n",
    "# YOUR CODE HERE\n",
    "\n",
    "# Complete feature list\n",
    "features = # YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6913331",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select the features of interest and the label\n",
    "diabetes_df = diabetes_df[features + [label]]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e7fcfc4",
   "metadata": {},
   "source": [
    "### 1.2. Train test split\n",
    "\n",
    "Use `train_test_split` to split the data into training and testing sets. Use `random_state=315` for reproducibility."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "105a5af5",
   "metadata": {},
   "outputs": [],
   "source": [
    "training_df, testing_df = # YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1981de6",
   "metadata": {},
   "source": [
    "### 1.3. Preprocess numerical features\n",
    "\n",
    "#### 1.3.1. Standard scale\n",
    "\n",
    "Neural networks perform better when features are scaled. Use `StandardScaler` to fit on the training features and transform both training and testing features.\n",
    "\n",
    "**Hint:** Fit the scaler on `training_df[numerical_features]`, then transform both `training_df[numerical_features]` and `testing_df[numerical_features]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "232a25d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_scaler = StandardScaler()\n",
    "\n",
    "# YOUR CODE HERE: fit and transform the numerical features"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44f79845",
   "metadata": {},
   "source": [
    "#### 1.3.2. Clip outliers with IQR method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65fa125c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR CODE HERE: implement IQR clipping for numerical features"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0d62a8a",
   "metadata": {},
   "source": [
    "### 1.4. Preprocess categorical features\n",
    "\n",
    "#### 1.4.1. Ordinal feature encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58b0061a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR CODE HERE: create and fit OrdinalEncoder, then transform both training and testing data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13ebb926",
   "metadata": {},
   "source": [
    "#### 1.4.2. Nominal feature encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7a5e92e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR CODE HERE: create and fit OneHotEncoder, transform features, and concatenate back to dataframes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "756b7139",
   "metadata": {},
   "source": [
    "#### 1.4.3. Update the feature list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68bcea8b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR CODE HERE: update the features list to include encoded features and remove the label"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ac64191",
   "metadata": {},
   "source": [
    "#### 1.4.4. Min/max scale categorical features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba67e611",
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR CODE HERE: create MinMaxScaler with feature_range=(-1, 1), fit on categorical features, and transform"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90b784ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "training_df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80005c89",
   "metadata": {},
   "source": [
    "## 2. Logistic regression model\n",
    "\n",
    "Logistic regression is a linear model for classification. It serves as a good baseline before trying more complex models like neural networks.\n",
    "\n",
    "### 2.1. Fit\n",
    "\n",
    "Create a `LogisticRegression` model and fit it on the training data. Use `max_iter=1000` to ensure convergence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca564d0f",
   "metadata": {},
   "outputs": [],
   "source": [
    "logistic_model = # YOUR CODE HERE\n",
    "fit_result = # YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "689f8ca7",
   "metadata": {},
   "source": [
    "### 2.2. Test set evaluation\n",
    "\n",
    "For classification, we can use accuracy, F1 score and/or AUC-ROC (and others) instead of R². Use sklearn's [`metrics`](https://scikit-learn.org/stable/api/sklearn.metrics.html) module ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22fa11d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "logistic_predictions = # YOUR CODE HERE\n",
    "logistic_accuracy = # YOUR CODE HERE\n",
    "logistic_f1 = # YOUR CODE HERE\n",
    "logistic_auc = # YOUR CODE HERE\n",
    "print(f'Logistic regression accuracy on test set: {logistic_accuracy:.4f}')\n",
    "print(f'Logistic regression F1 score on test set: {logistic_f1:.4f}')\n",
    "print(f'Logistic regression AUC-ROC score on test set: {logistic_auc:.4f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "919e06cf",
   "metadata": {},
   "source": [
    "### 2.3. Performance analysis\n",
    "\n",
    "For classification, visualize performance using a confusion matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c92bc5a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "879ce5e3",
   "metadata": {},
   "source": [
    "## 3. Multilayer perceptron (MLP) classifier\n",
    "\n",
    "Now let's build a neural network classifier using sklearn's [`MLPClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html).\n",
    "\n",
    "### 3.1. Single epoch training function\n",
    "\n",
    "Complete the training function below. It should:\n",
    "1. Split the data into training and validation sets\n",
    "2. Call `partial_fit` on the model (remember to pass `classes=[0, 1]` on the first call)\n",
    "3. Record training and validation [`log_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html) (aka binary cross-entropy) in the history dictionary\n",
    "\n",
    "**Hint:** Use `model.partial_fit(X, y, classes=[0, 1])` for the first epoch. For subsequent epochs, `partial_fit` remembers the classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "605c3197",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(model: MLPClassifier, df: pd.DataFrame, training_history: dict, classes: list = None) -> tuple[MLPClassifier, dict]:\n",
    "    '''Trains sklearn MLP classifier model on given dataframe using validation split.\n",
    "    Returns the updated model and training history dictionary containing training and\n",
    "    validation log loss. If classes are not provided, assumes 0 and 1.'''\n",
    "\n",
    "    global features, label\n",
    "\n",
    "    df, val_df = train_test_split(df, random_state=315)\n",
    "    \n",
    "    # YOUR CODE HERE: call partial_fit on the model\n",
    "    # If classes is provided, pass it to partial_fit\n",
    "    \n",
    "    # YOUR CODE HERE: append training and validation log loss to history\n",
    "    \n",
    "    return model, training_history"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17bf5072",
   "metadata": {},
   "source": [
    "### 3.2. Model training\n",
    "\n",
    "Create an `MLPClassifier` with:\n",
    "- `hidden_layer_sizes=(64, 32)` - two hidden layers\n",
    "- `activation='relu'` - ReLU activation function\n",
    "- `learning_rate_init=0.001` - initial learning rate\n",
    "- `warm_start=True` - keep weights between calls to fit\n",
    "- `random_state=315` - for reproducibility\n",
    "\n",
    "Train for 10 epochs using the training function above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "734cccf3",
   "metadata": {},
   "outputs": [],
   "source": [
    "epochs = 10\n",
    "\n",
    "training_history = {\n",
    "    'training_loss': [],\n",
    "    'validation_loss': []\n",
    "}\n",
    "\n",
    "mlp_model = # YOUR CODE HERE: create MLPClassifier\n",
    "\n",
    "for epoch in range(epochs):\n",
    "\n",
    "    # YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76005d01",
   "metadata": {},
   "source": [
    "### 3.3. Learning curves\n",
    "\n",
    "Plot the training and validation loss over epochs to visualize the learning process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7c94bd0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR CODE HERE: plot training and validation loss\n",
    "# Use plt.plot() for each curve\n",
    "# Add title, xlabel, ylabel, and legend"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2e947e3",
   "metadata": {},
   "source": [
    "### 3.4. Test set evaluation\n",
    "\n",
    "Evaluate the MLP model on the test set, similar to how you evaluated the logistic regression model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36bc02a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "mlp_predictions = # YOUR CODE HERE\n",
    "mlp_accuracy = # YOUR CODE HERE\n",
    "mlp_f1 = # YOUR CODE HERE\n",
    "mlp_auc = # YOUR CODE HERE\n",
    "print(f'MLP accuracy on test set: {mlp_accuracy:.4f}')\n",
    "print(f'MLP F1 score on test set: {mlp_f1:.4f}')\n",
    "print(f'MLP AUC-ROC score on test set: {mlp_auc:.4f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42cff8e0",
   "metadata": {},
   "source": [
    "### 3.5. Performance analysis\n",
    "\n",
    "Create a confusion matrix for the MLP model predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77851967",
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR CODE HERE: create confusion matrix for MLP predictions\n",
    "# Follow the same pattern as the logistic regression confusion matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40f1a0e4",
   "metadata": {},
   "source": [
    "## 4. Model comparison\n",
    "\n",
    "Compare the performance of both models side by side."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "809e2c9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'Logistic Regression accuracy on test set: {logistic_accuracy:.4f}')\n",
    "print(f'MLP accuracy on test set: {mlp_accuracy:.4f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bada64a",
   "metadata": {},
   "source": [
    "Create a side-by-side comparison of the confusion matrices for both models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d06ec0a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR CODE HERE"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}