{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "02ecf0d3",
   "metadata": {},
   "source": [
    "# Lesson 38: Text preprocessing pipeline activity\n",
    "\n",
    "In this activity, you will build a reusable text preprocessing pipeline and compare the effects of different preprocessing choices on text classification.\n",
    "\n",
    "1. **Custom preprocessing function** - Create a function that applies multiple preprocessing steps\n",
    "2. **Stemmer comparison** - Compare Porter, Lancaster, and Snowball stemmers\n",
    "3. **Classification impact** - Measure how preprocessing choices affect Naive Bayes performance\n",
    "\n",
    "## Notebook set-up\n",
    "\n",
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "043a1185",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "import nltk\n",
    "from nltk.corpus import movie_reviews, stopwords\n",
    "from nltk.stem import PorterStemmer, LancasterStemmer, SnowballStemmer\n",
    "from nltk.tokenize import word_tokenize\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "nltk.download('movie_reviews', quiet=True)\n",
    "nltk.download('punkt', quiet=True)\n",
    "nltk.download('punkt_tab', quiet=True)\n",
    "nltk.download('stopwords', quiet=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7423465",
   "metadata": {},
   "source": [
    "## 1. Load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "993a26fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load movie reviews\n",
    "documents = [\n",
    "    (' '.join(movie_reviews.words(fileid)), category)\n",
    "    for category in movie_reviews.categories()\n",
    "    for fileid in movie_reviews.fileids(category)\n",
    "]\n",
    "\n",
    "texts = [doc[0] for doc in documents]\n",
    "labels = [1 if doc[1] == 'pos' else 0 for doc in documents]\n",
    "\n",
    "print(f'Total documents: {len(texts)}')\n",
    "print(f'Positive reviews: {sum(labels)}')\n",
    "print(f'Negative reviews: {len(labels) - sum(labels)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "765b9fbf",
   "metadata": {},
   "source": [
    "## 2. Build preprocessing pipeline\n",
    "\n",
    "### Task 1: Create a reusable preprocessing function\n",
    "\n",
    "Complete the `preprocess_text` function below to apply the following steps:\n",
    "\n",
    "1. **Lowercase** the text\n",
    "2. **Remove punctuation** and numbers\n",
    "3. **Tokenize** the text\n",
    "4. **Remove stopwords**\n",
    "5. **Apply stemming** using the provided stemmer\n",
    "\n",
    "**Hints:**\n",
    "- Use `text.lower()` for lowercasing\n",
    "- Use `re.sub(r'[^a-z\\s]', '', text)` to keep only letters and spaces\n",
    "- Use `word_tokenize()` for tokenization\n",
    "- Filter tokens against `stopwords.words('english')`\n",
    "- Apply `stemmer.stem(token)` to each remaining token"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d5608a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "stop_words = set(stopwords.words('english'))\n",
    "\n",
    "def preprocess_text(text: str, stemmer=None) -> str:\n",
    "    '''Preprocess text by lowercasing, removing punctuation, tokenizing,\n",
    "    removing stopwords, and optionally stemming.\n",
    "    \n",
    "    TODO: Implement the preprocessing steps described above.\n",
    "    '''\n",
    "    \n",
    "    # Step 1: Lowercase\n",
    "    # YOUR CODE HERE\n",
    "    \n",
    "    # Step 2: Remove punctuation and numbers\n",
    "    # YOUR CODE HERE\n",
    "    \n",
    "    # Step 3: Tokenize\n",
    "    # YOUR CODE HERE\n",
    "    \n",
    "    # Step 4: Remove stopwords\n",
    "    # YOUR CODE HERE\n",
    "    \n",
    "    # Step 5: Apply stemming (if stemmer provided)\n",
    "    # YOUR CODE HERE\n",
    "    \n",
    "    # Join tokens back to string\n",
    "    return ' '.join(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "acc25bc8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test your preprocessing function\n",
    "sample_text = \"This movie was AMAZING! I loved the acting (10/10) and the storyline.\"\n",
    "porter = PorterStemmer()\n",
    "\n",
    "result = preprocess_text(sample_text, stemmer=porter)\n",
    "print(f'Original: {sample_text}')\n",
    "print(f'Preprocessed: {result}')\n",
    "\n",
    "# Expected output should be something like: \"movi amaz love act storylin\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "949061ff",
   "metadata": {},
   "source": [
    "## 3. Compare stemmers\n",
    "\n",
    "### Task 2: Compare stemmer outputs\n",
    "\n",
    "Use your preprocessing function to compare how different stemmers transform the same text. Run the cell below to see the differences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e388d5d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize stemmers\n",
    "stemmers = {\n",
    "    'Porter': PorterStemmer(),\n",
    "    'Lancaster': LancasterStemmer(),\n",
    "    'Snowball': SnowballStemmer('english'),\n",
    "    'None': None\n",
    "}\n",
    "\n",
    "# Compare on sample text\n",
    "sample = \"The running runners ran quickly through the beautiful forest\"\n",
    "\n",
    "print(f'Original: {sample}\\n')\n",
    "\n",
    "for name, stemmer in stemmers.items():\n",
    "    result = preprocess_text(sample, stemmer=stemmer)\n",
    "    print(f'{name:12s}: {result}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f85f48c7",
   "metadata": {},
   "source": [
    "## 4. Classification impact\n",
    "\n",
    "### Task 3: Measure preprocessing impact on classification\n",
    "\n",
    "Complete the code below to evaluate how different preprocessing choices affect Naive Bayes classification accuracy.\n",
    "\n",
    "**Your task:**\n",
    "1. Preprocess all texts using each stemmer configuration\n",
    "2. Train a Naive Bayes classifier for each\n",
    "3. Report accuracy for each configuration\n",
    "\n",
    "**Hints:**\n",
    "- Use a list comprehension to preprocess all texts: `[preprocess_text(t, stemmer) for t in texts]`\n",
    "- Use `train_test_split` with `random_state=315`\n",
    "- Use `CountVectorizer` to convert preprocessed texts to features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f58244e",
   "metadata": {},
   "outputs": [],
   "source": [
    "results = {}\n",
    "\n",
    "for name, stemmer in stemmers.items():\n",
    "\n",
    "    # TODO: Preprocess all texts using this stemmer\n",
    "    preprocessed_texts = None  # YOUR CODE HERE\n",
    "    \n",
    "    # TODO: Split into train/test sets (use random_state=315)\n",
    "    X_train, X_test, y_train, y_test = None, None, None, None  # YOUR CODE HERE\n",
    "    \n",
    "    # TODO: Vectorize using CountVectorizer\n",
    "    vectorizer = CountVectorizer()\n",
    "    # YOUR CODE HERE to fit_transform train and transform test\n",
    "    \n",
    "    # TODO: Train Naive Bayes classifier\n",
    "    # YOUR CODE HERE\n",
    "    \n",
    "    # TODO: Calculate accuracy on test set\n",
    "    accuracy = None  # YOUR CODE HERE\n",
    "    \n",
    "    results[name] = accuracy\n",
    "    print(f'{name:12s} stemmer accuracy: {accuracy:.4f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4829c17",
   "metadata": {},
   "source": [
    "## 5. Analysis questions\n",
    "\n",
    "After completing the tasks above, answer these questions:\n",
    "\n",
    "1. Which stemmer produced the highest accuracy? Why might this be?\n",
    "2. How does stemming affect the vocabulary size (number of unique words)?\n",
    "3. What are the trade-offs between aggressive stemming (Lancaster) and conservative stemming (Porter)?"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}