{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "783472ac",
   "metadata": {},
   "source": [
    "# Lesson 30 activity: hyperparameter optimization\n",
    "\n",
    "In the demo, we used Optuna to optimize just 2 hyperparameters:\n",
    "1. Number of convolutional blocks (1-4)\n",
    "2. Dropout rate (0.2-0.5)\n",
    "\n",
    "## Your Challenge\n",
    "\n",
    "Extend the optimization to include **additional hyperparameters**. The goal is to explore how different aspects of the model and training process affect performance.\n",
    "\n",
    "### Things to Consider\n",
    "\n",
    "Think about the different \"knobs\" you can tune in a neural network:\n",
    "\n",
    "- **Architecture choices**: How wide should the layers be? How many neurons in the fully connected layers?\n",
    "- **Training dynamics**: What about the learning rate? Could the optimizer itself be a choice?\n",
    "- **Regularization**: Are there other regularization techniques besides dropout?\n",
    "\n",
    "### Hints\n",
    "\n",
    "- Look at Optuna's `suggest_*` methods: `suggest_int()`, `suggest_float()`, `suggest_categorical()`\n",
    "- Some hyperparameters might need to be passed to `create_cnn()`, while others might be used when creating the optimizer\n",
    "- Be careful about search space sizes - more hyperparameters means more trials needed!\n",
    "- Consider using `suggest_float(..., log=True)` for hyperparameters that span orders of magnitude\n",
    "\n",
    "### Suggested Starting Points (pick 1-2 to add)\n",
    "\n",
    "1. **Learning rate**: What range makes sense? (Hint: think logarithmic scale)\n",
    "2. **Initial filters**: The demo uses 32 - what if you tried 16, 32, or 64?\n",
    "3. **FC layer sizes**: Could you optimize the fully connected layer dimensions?\n",
    "4. **Optimizer choice**: Adam vs SGD vs RMSprop - which works best?\n",
    "5. **Batch size**: Does this affect final accuracy?\n",
    "\n",
    "---\n",
    "\n",
    "## Setup\n",
    "\n",
    "To run this notebook install `cifar10_tools` via pip: `pip install cifar10_tools`\n",
    "\n",
    "**Note**: If you are not working in one of the course deeplearning containers, you will also need to pip install `optuna` and `optuna-dashboard` to run this notebook.\n",
    "\n",
    "```\n",
    "pip install optuna optuna-dashboard\n",
    "```\n",
    "\n",
    "The Optuna dashboard can be viewed either via the VS Code extension 'Optuna Dashboard', or via the built-in web server. Start it with:\n",
    "\n",
    "```\n",
    "optuna-dashboard sqlite:///data/simple_optimization.db --host 0.0.0.0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0658b7b8",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a70d6c42",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Standard library imports\n",
    "from pathlib import Path\n",
    "\n",
    "# Third-party imports\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import optuna\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import torchvision\n",
    "import torchvision.transforms as transforms\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "# Custom package imports\n",
    "from cifar10_tools.pytorch.training import train_model\n",
    "from cifar10_tools.pytorch.evaluation import evaluate_model\n",
    "from cifar10_tools.pytorch.plotting import (\n",
    "    plot_sample_images,\n",
    "    plot_learning_curves,\n",
    "    plot_confusion_matrix\n",
    ")\n",
    "\n",
    "# Suppress Optuna info messages (show only warnings and errors)\n",
    "optuna.logging.set_verbosity(optuna.logging.WARNING)\n",
    "\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "print(f'Using device: {device}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "301676f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fixed hyperparameters (some of these could become tunable!)\n",
    "batch_size = 1000\n",
    "initial_filters = 32\n",
    "fc_units_1 = 512\n",
    "fc_units_2 = 128\n",
    "use_batch_norm = True\n",
    "learning_rate = 0.001\n",
    "\n",
    "# Optuna settings\n",
    "n_trials = 30            # You may want more trials with more hyperparameters\n",
    "n_epochs_per_trial = 20  # Short training per trial\n",
    "n_epochs_final = 50      # Longer training for final model\n",
    "\n",
    "# Storage path for Optuna study - use a NEW name for your experiment!\n",
    "storage_path = Path('../data/activity_optimization.db')\n",
    "storage_path.parent.mkdir(parents=True, exist_ok=True)\n",
    "storage_url = f'sqlite:///{storage_path.resolve()}'\n",
    "\n",
    "# CIFAR-10 class names\n",
    "class_names = [\n",
    "    'airplane', 'automobile', 'bird', 'cat', 'deer', \n",
    "    'dog', 'frog', 'horse', 'ship', 'truck'\n",
    "]\n",
    "\n",
    "num_classes = len(class_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8b4dcd6",
   "metadata": {},
   "source": [
    "## 1. Load and Prepare Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc3ba5e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Data directory\n",
    "data_dir = Path('../data')\n",
    "data_dir.mkdir(parents=True, exist_ok=True)\n",
    "\n",
    "# Transform: convert to tensor and normalize RGB channels\n",
    "transform = transforms.Compose([\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))\n",
    "])\n",
    "\n",
    "# Download and load CIFAR-10\n",
    "train_dataset = torchvision.datasets.CIFAR10(\n",
    "    root=data_dir,\n",
    "    train=True,\n",
    "    download=True,\n",
    "    transform=transform\n",
    ")\n",
    "\n",
    "test_dataset = torchvision.datasets.CIFAR10(\n",
    "    root=data_dir,\n",
    "    train=False,\n",
    "    download=True,\n",
    "    transform=transform\n",
    ")\n",
    "\n",
    "print(f'Training samples: {len(train_dataset)}')\n",
    "print(f'Test samples: {len(test_dataset)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ab77bce",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Preload data to GPU for faster training\n",
    "X_train_full = torch.stack([img for img, _ in train_dataset]).to(device)\n",
    "y_train_full = torch.tensor([label for _, label in train_dataset]).to(device)\n",
    "X_test = torch.stack([img for img, _ in test_dataset]).to(device)\n",
    "y_test = torch.tensor([label for _, label in test_dataset]).to(device)\n",
    "\n",
    "# Split training data into train and validation sets (80/20)\n",
    "n_train = int(0.8 * len(X_train_full))\n",
    "indices = torch.randperm(len(X_train_full))\n",
    "\n",
    "X_train = X_train_full[indices[:n_train]]\n",
    "y_train = y_train_full[indices[:n_train]]\n",
    "X_val = X_train_full[indices[n_train:]]\n",
    "y_val = y_train_full[indices[n_train:]]\n",
    "\n",
    "print(f'X_train: {X_train.shape}')\n",
    "print(f'X_val: {X_val.shape}')\n",
    "print(f'X_test: {X_test.shape}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03a97db2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create DataLoaders\n",
    "train_tensor_dataset = torch.utils.data.TensorDataset(X_train, y_train)\n",
    "val_tensor_dataset = torch.utils.data.TensorDataset(X_val, y_val)\n",
    "test_tensor_dataset = torch.utils.data.TensorDataset(X_test, y_test)\n",
    "\n",
    "train_loader = DataLoader(train_tensor_dataset, batch_size=batch_size, shuffle=True)\n",
    "val_loader = DataLoader(val_tensor_dataset, batch_size=batch_size, shuffle=False)\n",
    "test_loader = DataLoader(test_tensor_dataset, batch_size=batch_size, shuffle=False)\n",
    "\n",
    "print(f'Training batches: {len(train_loader)}')\n",
    "print(f'Validation batches: {len(val_loader)}')\n",
    "print(f'Test batches: {len(test_loader)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0627967c",
   "metadata": {},
   "source": [
    "## 2. Define CNN Architecture\n",
    "\n",
    "**TODO**: Consider modifying this function to accept additional parameters.\n",
    "\n",
    "For example, you might want to make `initial_filters`, `fc_units_1`, or `fc_units_2` configurable.\n",
    "\n",
    "*Think about: What would need to change in the function signature and body?*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4931ddd",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_cnn(n_conv_blocks: int, dropout_rate: float) -> nn.Sequential:\n",
    "    '''Create a CNN with configurable architecture.\n",
    "    \n",
    "    Args:\n",
    "        n_conv_blocks: Number of convolutional blocks (1-4)\n",
    "        dropout_rate: Dropout probability\n",
    "        \n",
    "        # TODO: Add more parameters here if needed!\n",
    "    \n",
    "    Returns:\n",
    "        nn.Sequential model\n",
    "    '''\n",
    "\n",
    "    layers = []\n",
    "    in_channels = 3  # RGB input\n",
    "    current_size = 32  # Input image size\n",
    "    \n",
    "    for block_idx in range(n_conv_blocks):\n",
    "        out_channels = initial_filters * (2 ** block_idx)\n",
    "        \n",
    "        # Conv -> BatchNorm -> ReLU -> Conv -> BatchNorm -> ReLU -> Pool -> Dropout\n",
    "        layers.append(nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1))\n",
    "        layers.append(nn.BatchNorm2d(out_channels))\n",
    "        layers.append(nn.ReLU())\n",
    "        \n",
    "        layers.append(nn.Conv2d(out_channels, out_channels, kernel_size=3, padding=1))\n",
    "        layers.append(nn.BatchNorm2d(out_channels))\n",
    "        layers.append(nn.ReLU())\n",
    "        \n",
    "        layers.append(nn.MaxPool2d(2, 2))\n",
    "        layers.append(nn.Dropout(dropout_rate))\n",
    "        \n",
    "        in_channels = out_channels\n",
    "        current_size //= 2\n",
    "    \n",
    "    # Calculate flattened size\n",
    "    final_channels = initial_filters * (2 ** (n_conv_blocks - 1))\n",
    "    flattened_size = final_channels * current_size * current_size\n",
    "    \n",
    "    # Classifier (3 fully connected layers)\n",
    "    layers.append(nn.Flatten())\n",
    "    layers.append(nn.Linear(flattened_size, fc_units_1))\n",
    "    layers.append(nn.ReLU())\n",
    "    layers.append(nn.Dropout(dropout_rate))\n",
    "    layers.append(nn.Linear(fc_units_1, fc_units_2))\n",
    "    layers.append(nn.ReLU())\n",
    "    layers.append(nn.Dropout(dropout_rate))\n",
    "    layers.append(nn.Linear(fc_units_2, num_classes))\n",
    "    \n",
    "    return nn.Sequential(*layers)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2ba1a3e",
   "metadata": {},
   "source": [
    "## 3. Optuna Hyperparameter Optimization\n",
    "\n",
    "### 3.1. Training Function for Trials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f363238b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_trial(\n",
    "    model: nn.Module,\n",
    "    optimizer: optim.Optimizer,\n",
    "    criterion: nn.Module,\n",
    "    train_loader: DataLoader,\n",
    "    val_loader: DataLoader,\n",
    "    n_epochs: int,\n",
    "    trial: optuna.Trial\n",
    ") -> float:\n",
    "    '''Train a model for a single Optuna trial with pruning support.'''\n",
    "    \n",
    "    best_val_accuracy = 0.0\n",
    "    \n",
    "    for epoch in range(n_epochs):\n",
    "\n",
    "        # Training phase\n",
    "        model.train()\n",
    "\n",
    "        for images, labels in train_loader:\n",
    "            optimizer.zero_grad()\n",
    "            outputs = model(images)\n",
    "            loss = criterion(outputs, labels)\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "        \n",
    "        # Validation phase\n",
    "        model.eval()\n",
    "        val_correct = 0\n",
    "        val_total = 0\n",
    "        \n",
    "        with torch.no_grad():\n",
    "            for images, labels in val_loader:\n",
    "\n",
    "                outputs = model(images)\n",
    "                _, predicted = torch.max(outputs.data, 1)\n",
    "                val_total += labels.size(0)\n",
    "                val_correct += (predicted == labels).sum().item()\n",
    "        \n",
    "        val_accuracy = 100 * val_correct / val_total\n",
    "        best_val_accuracy = max(best_val_accuracy, val_accuracy)\n",
    "        \n",
    "        # Report for pruning\n",
    "        trial.report(val_accuracy, epoch)\n",
    "        \n",
    "        if trial.should_prune():\n",
    "            raise optuna.TrialPruned()\n",
    "    \n",
    "    return best_val_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f38da50d",
   "metadata": {},
   "source": [
    "### 3.2. Define Objective Function\n",
    "\n",
    "**TODO**: This is where you add your new hyperparameters!\n",
    "\n",
    "Currently we optimize:\n",
    "- `n_conv_blocks`: 1 to 4 convolutional blocks\n",
    "- `dropout_rate`: 0.2 to 0.5\n",
    "\n",
    "**Your task**: Add at least 1-2 more hyperparameters to optimize.\n",
    "\n",
    "#### Useful Optuna methods:\n",
    "```python\n",
    "# For integers (e.g., number of filters, layer sizes)\n",
    "trial.suggest_int('param_name', low, high)\n",
    "\n",
    "# For floats (e.g., dropout, learning rate)\n",
    "trial.suggest_float('param_name', low, high)\n",
    "\n",
    "# For floats on log scale (great for learning rates!)\n",
    "trial.suggest_float('param_name', low, high, log=True)\n",
    "\n",
    "# For categorical choices (e.g., optimizer type)\n",
    "trial.suggest_categorical('param_name', ['option1', 'option2', 'option3'])\n",
    "```\n",
    "\n",
    "#### Example additions you might try:\n",
    "```python\n",
    "# Optimize learning rate (log scale is important here!)\n",
    "lr = trial.suggest_float('learning_rate', 1e-5, 1e-2, log=True)\n",
    "\n",
    "# Optimize initial filter count\n",
    "init_filters = trial.suggest_categorical('initial_filters', [16, 32, 64])\n",
    "\n",
    "# Optimize optimizer choice\n",
    "optimizer_name = trial.suggest_categorical('optimizer', ['Adam', 'SGD', 'RMSprop'])\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d8fe851",
   "metadata": {},
   "outputs": [],
   "source": [
    "def objective(trial: optuna.Trial) -> float:\n",
    "    '''Optuna objective function.\n",
    "    \n",
    "    TODO: Add more hyperparameters to optimize!\n",
    "    '''\n",
    "    \n",
    "    # === Existing hyperparameters ===\n",
    "    n_conv_blocks = trial.suggest_int('n_conv_blocks', 1, 4)\n",
    "    dropout_rate = trial.suggest_float('dropout_rate', 0.2, 0.5)\n",
    "    \n",
    "    # === TODO: Add your new hyperparameters here! ===\n",
    "    # Example:\n",
    "    # lr = trial.suggest_float('learning_rate', 1e-5, 1e-2, log=True)\n",
    "    \n",
    "    \n",
    "    # Create model\n",
    "    model = create_cnn(\n",
    "        n_conv_blocks=n_conv_blocks,\n",
    "        dropout_rate=dropout_rate\n",
    "        # TODO: Pass any new architecture parameters here\n",
    "    ).to(device)\n",
    "    \n",
    "    # Create optimizer\n",
    "    # TODO: If you're optimizing learning rate or optimizer type,\n",
    "    #       you'll need to modify this section!\n",
    "    optimizer = optim.Adam(model.parameters(), lr=learning_rate)\n",
    "    \n",
    "    # Set loss function\n",
    "    criterion = nn.CrossEntropyLoss()\n",
    "    \n",
    "    # Train and return validation accuracy\n",
    "    try:\n",
    "        return train_trial(\n",
    "            model=model,\n",
    "            optimizer=optimizer,\n",
    "            criterion=criterion,\n",
    "            train_loader=train_loader,\n",
    "            val_loader=val_loader,\n",
    "            n_epochs=n_epochs_per_trial,\n",
    "            trial=trial\n",
    "        )\n",
    "\n",
    "    except torch.cuda.OutOfMemoryError:\n",
    "\n",
    "        torch.cuda.empty_cache()\n",
    "        raise optuna.TrialPruned('CUDA OOM')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22eabcbd",
   "metadata": {},
   "source": [
    "### 3.3. Run Optimization\n",
    "\n",
    "**Note**: With more hyperparameters, you may want to increase `n_trials` for better exploration of the search space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf876922",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "    \n",
    "# Create Optuna study (maximize validation accuracy)\n",
    "study = optuna.create_study(\n",
    "    direction='maximize',\n",
    "    study_name='activity_optimization',\n",
    "    storage=storage_url,\n",
    "    load_if_exists=True,\n",
    "    pruner=optuna.pruners.MedianPruner(n_warmup_steps=3)\n",
    ")\n",
    "\n",
    "# Run optimization\n",
    "study.optimize(objective, n_trials=n_trials, show_progress_bar=True)\n",
    "\n",
    "print(f'Best validation accuracy: {study.best_trial.value:.2f}%')\n",
    "print(f'\\nBest hyperparameters:')\n",
    "\n",
    "for key, value in study.best_trial.params.items():\n",
    "    print(f'  {key}: {value}')\n",
    "\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80ed4c80",
   "metadata": {},
   "source": [
    "### 3.4. Visualize Optimization Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c8658d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(1, 2, figsize=(12, 4))\n",
    "\n",
    "# Optimization history\n",
    "axes[0].set_title('Optimization History')\n",
    "\n",
    "trial_numbers = [t.number for t in study.trials if t.value is not None]\n",
    "trial_values = [t.value for t in study.trials if t.value is not None]\n",
    "\n",
    "axes[0].plot(trial_numbers, trial_values, 'ko-', alpha=0.6)\n",
    "axes[0].axhline(y=study.best_value, color='r', linestyle='--', label=f'Best: {study.best_value:.2f}%')\n",
    "axes[0].set_xlabel('Trial')\n",
    "axes[0].set_ylabel('Validation Accuracy (%)')\n",
    "axes[0].legend()\n",
    "\n",
    "# Hyperparameter importance\n",
    "axes[1].set_title('Hyperparameter Importance')\n",
    "completed_trials = [t for t in study.trials if t.state == optuna.trial.TrialState.COMPLETE]\n",
    "\n",
    "if len(completed_trials) >= 5:\n",
    "    importance = optuna.importance.get_param_importances(study)\n",
    "    params = list(importance.keys())\n",
    "    values = list(importance.values())\n",
    "    axes[1].barh(params, values, color='steelblue')\n",
    "    axes[1].set_xlabel('Importance')\n",
    "\n",
    "else:\n",
    "    axes[1].text(0.5, 0.5, 'Not enough trials\\nfor importance analysis', \n",
    "                 ha='center', va='center', transform=axes[1].transAxes)\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c4c604a",
   "metadata": {},
   "source": [
    "## 4. Train Final Model with Best Hyperparameters\n",
    "\n",
    "**TODO**: Update this section to use any new hyperparameters you added!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7bfb906e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get best hyperparameters\n",
    "best_params = study.best_trial.params\n",
    "\n",
    "print('Best hyperparameters:')\n",
    "for key, value in best_params.items():\n",
    "    print(f'  {key}: {value}')\n",
    "\n",
    "# Create model with best hyperparameters\n",
    "# TODO: Pass any new parameters you added!\n",
    "best_model = create_cnn(\n",
    "    n_conv_blocks=best_params['n_conv_blocks'],\n",
    "    dropout_rate=best_params['dropout_rate']\n",
    ").to(device)\n",
    "\n",
    "# Create optimizer\n",
    "# TODO: Use best learning rate / optimizer if you optimized those!\n",
    "best_optimizer = optim.Adam(best_model.parameters(), lr=learning_rate)\n",
    "\n",
    "# Set loss function\n",
    "criterion = nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac27bcea",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "# Train for more epochs\n",
    "history = train_model(\n",
    "    model=best_model,\n",
    "    train_loader=train_loader,\n",
    "    val_loader=val_loader,\n",
    "    criterion=criterion,\n",
    "    optimizer=best_optimizer,\n",
    "    epochs=n_epochs_final,\n",
    "    print_every=10\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4fcaedde",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot learning curves\n",
    "fig, axes = plot_learning_curves(history)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59457bf4",
   "metadata": {},
   "source": [
    "## 5. Evaluate on Test Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2020fa23",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_accuracy, predictions, true_labels = evaluate_model(best_model, test_loader)\n",
    "print(f'Test accuracy: {test_accuracy:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df1f1f38",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Confusion matrix\n",
    "fig, ax = plot_confusion_matrix(true_labels, predictions, class_names)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa3aa8db",
   "metadata": {},
   "source": [
    "## 6. Reflection Questions\n",
    "\n",
    "After completing your optimization, answer these questions:\n",
    "\n",
    "1. **Which hyperparameters did you add?** Why did you choose those?\n",
    "\n",
    "2. **Which hyperparameter was most important** according to Optuna's importance analysis?\n",
    "\n",
    "3. **Did adding more hyperparameters improve your best accuracy** compared to the demo's 2-parameter search?\n",
    "\n",
    "4. **What challenges did you encounter** when expanding the search space?\n",
    "\n",
    "5. **If you had more time/compute, what else would you try?**\n",
    "\n",
    "*Your answers here:*\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}