{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b9d80ae0",
   "metadata": {},
   "source": [
    "# Lesson 39: TF-IDF parameter tuning activity (solution)\n",
    "\n",
    "In this activity, you will explore how different TF-IDF parameters affect text classification performance.\n",
    "\n",
    "1. **N-gram range** - Compare unigrams, bigrams, and trigrams\n",
    "2. **Vocabulary size** - Explore the effect of max_features\n",
    "3. **Document frequency filtering** - Use min_df and max_df parameters\n",
    "\n",
    "## Notebook set-up\n",
    "\n",
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2f6aba9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk.corpus import movie_reviews\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "nltk.download('movie_reviews', quiet=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9cb9c38",
   "metadata": {},
   "source": [
    "## 1. Load and prepare data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d610510",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load movie reviews\n",
    "documents = [\n",
    "    (' '.join(movie_reviews.words(fileid)), category)\n",
    "    for category in movie_reviews.categories()\n",
    "    for fileid in movie_reviews.fileids(category)\n",
    "]\n",
    "\n",
    "texts = [doc[0] for doc in documents]\n",
    "labels = [1 if doc[1] == 'pos' else 0 for doc in documents]\n",
    "\n",
    "# Split data\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    texts, labels, test_size=0.2, random_state=315\n",
    ")\n",
    "\n",
    "print(f'Training samples: {len(X_train)}')\n",
    "print(f'Testing samples: {len(X_test)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4084d4f3",
   "metadata": {},
   "source": [
    "## 2. Helper function\n",
    "\n",
    "This function trains and evaluates a classifier with a given vectorizer configuration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5284d4d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_vectorizer(vectorizer, X_train, X_test, y_train, y_test):\n",
    "    '''Train Naive Bayes with given vectorizer and return accuracy and vocab size.'''\n",
    "    \n",
    "    X_train_vec = vectorizer.fit_transform(X_train)\n",
    "    X_test_vec = vectorizer.transform(X_test)\n",
    "    \n",
    "    model = MultinomialNB()\n",
    "    model.fit(X_train_vec, y_train)\n",
    "    predictions = model.predict(X_test_vec)\n",
    "    \n",
    "    accuracy = accuracy_score(y_test, predictions)\n",
    "    vocab_size = len(vectorizer.vocabulary_)\n",
    "    \n",
    "    return accuracy, vocab_size"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c5dd0a4",
   "metadata": {},
   "source": [
    "## 3. N-gram range experiment\n",
    "\n",
    "### Task 1: Compare unigrams, bigrams, and combinations\n",
    "\n",
    "Complete the code below to test different n-gram ranges and observe their effect on accuracy.\n",
    "\n",
    "**Hints:**\n",
    "- `ngram_range=(1, 1)` uses only unigrams (single words)\n",
    "- `ngram_range=(1, 2)` uses unigrams and bigrams\n",
    "- `ngram_range=(2, 2)` uses only bigrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a7f4d52",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define n-gram configurations to test\n",
    "ngram_configs = [\n",
    "    (1, 1),  # Unigrams only\n",
    "    (1, 2),  # Unigrams and bigrams\n",
    "    (2, 2),  # Bigrams only\n",
    "    (1, 3),  # Unigrams, bigrams, and trigrams\n",
    "]\n",
    "\n",
    "print('N-gram range comparison:\\n')\n",
    "for ngram_range in ngram_configs:\n",
    "\n",
    "    # Create TfidfVectorizer with this ngram_range\n",
    "    vectorizer = TfidfVectorizer(ngram_range=ngram_range)\n",
    "    \n",
    "    # Evaluate\n",
    "    accuracy, vocab_size = evaluate_vectorizer(\n",
    "        vectorizer, X_train, X_test, y_train, y_test\n",
    "    )\n",
    "    \n",
    "    print(f'ngram_range={ngram_range}: accuracy={accuracy:.4f}, vocab_size={vocab_size:,}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a04039c",
   "metadata": {},
   "source": [
    "## 4. Vocabulary size experiment\n",
    "\n",
    "### Task 2: Explore max_features parameter\n",
    "\n",
    "The `max_features` parameter limits vocabulary to the top N features. Test how this affects accuracy.\n",
    "\n",
    "**Hints:**\n",
    "- Try values like 100, 500, 1000, 5000, 10000, None (unlimited)\n",
    "- Consider the trade-off between model size and accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74f635d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define max_features values to test\n",
    "max_features_values = [\n",
    "    100,\n",
    "    500,\n",
    "    1000,\n",
    "    5000,\n",
    "    10000,\n",
    "    None,  # Unlimited\n",
    "]\n",
    "\n",
    "print('max_features comparison:\\n')\n",
    "for max_features in max_features_values:\n",
    "\n",
    "    # Create TfidfVectorizer with this max_features\n",
    "    vectorizer = TfidfVectorizer(max_features=max_features)\n",
    "    \n",
    "    # Evaluate\n",
    "    accuracy, vocab_size = evaluate_vectorizer(\n",
    "        vectorizer, X_train, X_test, y_train, y_test\n",
    "    )\n",
    "    \n",
    "    print(f'max_features={str(max_features):>6s}: accuracy={accuracy:.4f}, vocab_size={vocab_size:,}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "553d75a6",
   "metadata": {},
   "source": [
    "## 5. Document frequency filtering\n",
    "\n",
    "### Task 3: Use min_df and max_df to filter terms\n",
    "\n",
    "- `min_df` removes terms appearing in fewer than N documents (or proportion)\n",
    "- `max_df` removes terms appearing in more than N documents (or proportion)\n",
    "\n",
    "**Your task:** Experiment with different combinations and observe the effect.\n",
    "\n",
    "**Hints:**\n",
    "- `min_df=5` means \"must appear in at least 5 documents\"\n",
    "- `max_df=0.95` means \"must appear in at most 95% of documents\"\n",
    "- These help remove rare typos and very common words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09564827",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define min_df/max_df configurations to test\n",
    "df_configs = [\n",
    "    (1, 1.0),    # No filtering\n",
    "    (5, 1.0),    # Remove rare terms only\n",
    "    (1, 0.95),   # Remove very common terms only\n",
    "    (5, 0.95),   # Remove both rare and common terms\n",
    "    (10, 0.8),   # More aggressive filtering\n",
    "    (10, 0.5),   # Very aggressive filtering\n",
    "]\n",
    "\n",
    "print('Document frequency filtering comparison:\\n')\n",
    "for min_df, max_df in df_configs:\n",
    "\n",
    "    # Create TfidfVectorizer with these parameters\n",
    "    vectorizer = TfidfVectorizer(min_df=min_df, max_df=max_df)\n",
    "    \n",
    "    # Evaluate\n",
    "    accuracy, vocab_size = evaluate_vectorizer(\n",
    "        vectorizer, X_train, X_test, y_train, y_test\n",
    "    )\n",
    "    \n",
    "    print(f'min_df={min_df}, max_df={max_df}: accuracy={accuracy:.4f}, vocab_size={vocab_size:,}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e744147",
   "metadata": {},
   "source": [
    "## 6. Analysis questions\n",
    "\n",
    "After completing the experiments above, answer these questions:\n",
    "\n",
    "1. Which n-gram configuration gave the best accuracy? Why?\n",
    "2. At what point does reducing max_features start hurting accuracy significantly?\n",
    "3. How do min_df and max_df affect both vocabulary size and accuracy?\n",
    "4. What combination of parameters would you recommend for a production system?"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}