{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "052b23ff",
   "metadata": {},
   "source": [
    "# Lesson 38: Text preprocessing pipeline activity solution\n",
    "\n",
    "In this activity, you will build a reusable text preprocessing pipeline and compare the effects of different preprocessing choices on text classification.\n",
    "\n",
    "1. **Custom preprocessing function** - Create a function that applies multiple preprocessing steps\n",
    "2. **Stemmer comparison** - Compare Porter, Lancaster, and Snowball stemmers\n",
    "3. **Classification impact** - Measure how preprocessing choices affect Naive Bayes performance\n",
    "\n",
    "## Notebook set-up\n",
    "\n",
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d0931aac",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "import nltk\n",
    "from nltk.corpus import movie_reviews, stopwords\n",
    "from nltk.stem import PorterStemmer, LancasterStemmer, SnowballStemmer, WordNetLemmatizer\n",
    "from nltk.tokenize import word_tokenize\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "nltk.download('movie_reviews', quiet=True)\n",
    "nltk.download('punkt', quiet=True)\n",
    "nltk.download('punkt_tab', quiet=True)\n",
    "nltk.download('stopwords', quiet=True)\n",
    "nltk.download('wordnet', quiet=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "369e7598",
   "metadata": {},
   "source": [
    "## 1. Load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b556a9f1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total documents: 2000\n",
      "Positive reviews: 1000\n",
      "Negative reviews: 1000\n"
     ]
    }
   ],
   "source": [
    "# Load movie reviews\n",
    "documents = [\n",
    "    (' '.join(movie_reviews.words(fileid)), category)\n",
    "    for category in movie_reviews.categories()\n",
    "    for fileid in movie_reviews.fileids(category)\n",
    "]\n",
    "\n",
    "texts = [doc[0] for doc in documents]\n",
    "labels = [1 if doc[1] == 'pos' else 0 for doc in documents]\n",
    "\n",
    "print(f'Total documents: {len(texts)}')\n",
    "print(f'Positive reviews: {sum(labels)}')\n",
    "print(f'Negative reviews: {len(labels) - sum(labels)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7de36e78",
   "metadata": {},
   "source": [
    "## 2. Build preprocessing pipeline\n",
    "\n",
    "### Task 1: Create a reusable preprocessing function\n",
    "\n",
    "Complete the `preprocess_text` function below to apply the following steps:\n",
    "\n",
    "1. **Lowercase** the text\n",
    "2. **Remove punctuation** and numbers\n",
    "3. **Tokenize** the text\n",
    "4. **Remove stopwords**\n",
    "5. **Apply stemming** using the provided stemmer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "46facb6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "stop_words = set(stopwords.words('english'))\n",
    "\n",
    "def preprocess_text(text: str, reducer=None) -> str:\n",
    "    '''Preprocess text by lowercasing, removing punctuation, tokenizing,\n",
    "    removing stopwords, and optionally stemming/lemmatizing.\n",
    "    '''\n",
    "\n",
    "    # Step 1: Lowercase\n",
    "    text = text.lower()\n",
    "\n",
    "    # Step 2: Remove punctuation and numbers\n",
    "    text = re.sub(r'[^a-z\\s]', '', text)\n",
    "\n",
    "    # Step 3: Tokenize\n",
    "    tokens = word_tokenize(text)\n",
    "\n",
    "    # Step 4: Remove stopwords\n",
    "    tokens = [t for t in tokens if t not in stop_words]\n",
    "\n",
    "    # Step 5: Apply stemming or lemmatization (if reducer provided)\n",
    "    if reducer:\n",
    "        if isinstance(reducer, WordNetLemmatizer):\n",
    "            tokens = [reducer.lemmatize(t) for t in tokens]\n",
    "\n",
    "        else:\n",
    "            tokens = [reducer.stem(t) for t in tokens]\n",
    "\n",
    "    # Join tokens back to string\n",
    "    return ' '.join(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ffd284b6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original: This movie was AMAZING! I loved the acting (10/10) and the storyline.\n",
      "Preprocessed: movi amaz love act storylin\n"
     ]
    }
   ],
   "source": [
    "# Test your preprocessing function\n",
    "sample_text = \"This movie was AMAZING! I loved the acting (10/10) and the storyline.\"\n",
    "porter = PorterStemmer()\n",
    "\n",
    "result = preprocess_text(sample_text, reducer=porter)\n",
    "print(f'Original: {sample_text}')\n",
    "print(f'Preprocessed: {result}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0159d4ba",
   "metadata": {},
   "source": [
    "## 3. Compare stemmers and lemmatizers\n",
    "\n",
    "### Task 2: Compare stemmer and lemmatizer outputs\n",
    "\n",
    "Use your preprocessing function to compare how different stemmers and lemmatizers transform the same text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ec912020",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original: The running runners ran quickly through the beautiful forest\n",
      "\n",
      "Porter      : run runner ran quickli beauti forest\n",
      "Lancaster   : run run ran quick beauty forest\n",
      "Snowball    : run runner ran quick beauti forest\n",
      "WordNet     : running runner ran quickly beautiful forest\n",
      "None        : running runners ran quickly beautiful forest\n"
     ]
    }
   ],
   "source": [
    "# Initialize stemmers and lemmatizers\n",
    "reducers = {\n",
    "    'Porter': PorterStemmer(),\n",
    "    'Lancaster': LancasterStemmer(),\n",
    "    'Snowball': SnowballStemmer('english'),\n",
    "    'WordNet': WordNetLemmatizer(),\n",
    "    'None': None\n",
    "}\n",
    "\n",
    "# Compare on sample text\n",
    "sample = \"The running runners ran quickly through the beautiful forest\"\n",
    "\n",
    "print(f'Original: {sample}\\n')\n",
    "\n",
    "for name, reducer in reducers.items():\n",
    "    result = preprocess_text(sample, reducer=reducer)\n",
    "    print(f'{name:12s}: {result}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b18590fa",
   "metadata": {},
   "source": [
    "## 4. Classification impact\n",
    "\n",
    "### Task 3: Measure preprocessing impact on classification\n",
    "\n",
    "Evaluate how different preprocessing choices affect Naive Bayes classification accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "45ee7961",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Porter       accuracy: 0.7925\n",
      "Lancaster    accuracy: 0.7775\n",
      "Snowball     accuracy: 0.8025\n",
      "WordNet      accuracy: 0.8000\n",
      "None         accuracy: 0.7950\n"
     ]
    }
   ],
   "source": [
    "results = {}\n",
    "\n",
    "for name, reducer in reducers.items():\n",
    "\n",
    "    # Preprocess all texts using this stemmer/lemmatizer\n",
    "    preprocessed_texts = [preprocess_text(t, reducer) for t in texts]\n",
    "\n",
    "    # Split into train/test sets\n",
    "    X_train, X_test, y_train, y_test = train_test_split(\n",
    "        preprocessed_texts,\n",
    "        labels,\n",
    "        test_size=0.2,\n",
    "        random_state=315\n",
    "    )\n",
    "\n",
    "    # Vectorize using CountVectorizer\n",
    "    vectorizer = CountVectorizer()\n",
    "    X_train_vec = vectorizer.fit_transform(X_train)\n",
    "    X_test_vec = vectorizer.transform(X_test)\n",
    "\n",
    "    # Train Naive Bayes classifier\n",
    "    clf = MultinomialNB()\n",
    "    clf.fit(X_train_vec, y_train)\n",
    "\n",
    "    # Calculate accuracy on test set\n",
    "    y_pred = clf.predict(X_test_vec)\n",
    "    accuracy = accuracy_score(y_test, y_pred)\n",
    "\n",
    "    results[name] = accuracy\n",
    "    print(f'{name:12s} accuracy: {accuracy:.4f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fae9680",
   "metadata": {},
   "source": [
    "## 5. Analysis questions\n",
    "\n",
    "**1. Which approach produced the highest accuracy? Why might this be?**\n",
    "\n",
    "The results may vary slightly, but typically no reduction, lemmatization, or conservative stemming (Porter/Snowball) produces similar or slightly better accuracy than aggressive stemming. This is because aggressive stemming can conflate words with different meanings (e.g., \"universal\" and \"university\" both become \"univers\" with Lancaster), introducing noise into the features. Lemmatization often performs well because it produces valid dictionary words while still reducing vocabulary.\n",
    "\n",
    "**2. How does stemming/lemmatization affect the vocabulary size (number of unique words)?**\n",
    "\n",
    "Both stemming and lemmatization reduce vocabulary size by collapsing different word forms. More aggressive stemmers (Lancaster) create smaller vocabularies than conservative ones (Porter). Lemmatization typically produces a vocabulary between no reduction and Porter stemming. We can measure this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "12e9cec2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Porter      : 25,322 unique terms\n",
      "Lancaster   : 20,665 unique terms\n",
      "Snowball    : 24,838 unique terms\n",
      "WordNet     : 34,402 unique terms\n",
      "None        : 38,786 unique terms\n"
     ]
    }
   ],
   "source": [
    "# Measure vocabulary size for each configuration\n",
    "for name, reducer in reducers.items():\n",
    "\n",
    "    preprocessed_texts = [preprocess_text(t, reducer) for t in texts]\n",
    "    vectorizer = CountVectorizer()\n",
    "    vectorizer.fit(preprocessed_texts)\n",
    "    vocab_size = len(vectorizer.vocabulary_)\n",
    "    print(f'{name:12s}: {vocab_size:,} unique terms')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60a13b3c",
   "metadata": {},
   "source": [
    "**3. What are the trade-offs between stemming and lemmatization?**\n",
    "\n",
    "| Aspect | Aggressive Stemming (Lancaster) | Conservative Stemming (Porter) | Lemmatization (WordNet) |\n",
    "|--------|--------------------------------|-------------------------------|-------------------------|\n",
    "| Vocabulary size | Smallest | Medium | Medium-Large |\n",
    "| Output validity | Non-words | Non-words | Valid words |\n",
    "| Conflation errors | More frequent | Less frequent | Least frequent |\n",
    "| Computational cost | Fast | Fast | Slower (dictionary lookup) |\n",
    "| Context awareness | None | None | Limited (POS tagging helps) |\n",
    "\n",
    "**When to use each:**\n",
    "- **Aggressive stemming**: When recall is important and vocabulary reduction is critical (e.g., limited memory)\n",
    "- **Conservative stemming**: When you need speed and moderate vocabulary reduction\n",
    "- **Lemmatization**: When interpretability matters or downstream tasks need valid words\n",
    "- **No reduction**: When exact word forms carry important meaning or when using modern embeddings"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}