{ "cells": [ { "cell_type": "markdown", "id": "3a8a2e23", "metadata": {}, "source": [ "# Activity SOLUTION: Improving sentiment analysis with convolutional layers\n", "\n", "**Note**: this notebook is part of the complete lesson 34 repository [`gperdrizet/RNNs`](https://github.com/gperdrizet/RNNs) and is intended to be run in the environment set-up by that repository.\n", "\n", "> **Note**: This is the solution notebook. Compare your implementation with this one after completing the activity.\n", "\n", "## Objective\n", "\n", "In this activity, you will extend the baseline RNN sentiment classifier by adding **convolutional layers** to create a hybrid CNN-RNN architecture. Convolutional layers can capture local n-gram patterns (like \"not good\" or \"very happy\") before the recurrent layers process the sequence.\n", "\n", "**Your tasks:**\n", "1. Run the baseline GRU model and note its performance\n", "2. Modify the model architecture to include convolutional layers\n", "3. Train your CNN-RNN hybrid model\n", "4. Compare the results to the baseline\n", "\n", "---\n", "\n", "## Key terms\n", "\n", "- **Tokenization**: Splitting text into individual units (tokens), typically words. For example, \"I love this!\" becomes `[\"i\", \"love\", \"this\"]`. This is the first step in converting text to a format neural networks can process.\n", "\n", "- **Embedding**: A dense vector representation of a token. Instead of one-hot encoding (sparse, high-dimensional), embeddings map each word to a fixed-size vector (e.g., 100 dimensions) where similar words have similar vectors. We use pretrained GloVe embeddings trained on Twitter data.\n", "\n", "- **LSTM/GRU**: Variants of RNNs designed to handle long sequences better than SimpleRNN. They use \"gates\" to control information flow, solving the vanishing gradient problem. **GRU** (Gated Recurrent Unit) has 2 gates and is simpler; **LSTM** (Long Short-Term Memory) has 3 gates and more parameters. Both work well in practice — we use GRU here for efficiency.\n", "\n", "## How the GRU processes a sequence\n", "\n", "Given input sequence `[\"i\", \"love\", \"this]`, the GRU predicts sentiment:\n", "\n", "```text\n", "\n", " ┌────────────◀───────────┐ 'h₁', then 'h₂'\n", " ▼ │\n", " ╔══════════════════════════════╗\n", " ║ │ GRU ▲ ║\n", " ║ │ │ ║\n", " 'I' then, ║ ├──▶Reset───▶candidate──▶│ ║ ╔═══════╗ \n", "'love', then ══▶║ │ gate hidden │ ╠══'h₃'══▶║ Dense ╠══▶ Sentiment\n", " 'this' ║ │ state │ ║ ╚═══════╝\n", " ║ │ │ ║\n", " ║ └─────▶Update gate───────┘ ║\n", " ╚══════════════════════════════╝\n", "\n", "```\n", "\n", "- **Reset gate**: Controls how much the prior hidden state influences the new candidate hidden state. Gates are **dynamic weights** (0-1) — one per hidden unit — computed fresh for each input. Depend on **both** the previous hidden state AND the current input.\n", "- **Update gate**: Controls how much prior hidden state vs new candidate hidden state goes into the new hidden state. Also computed from previous hidden state + current input, just with different learned weights.\n", "- **Bidirectional recurrent layers**: Processes a sequence in both directions (forward and backward) and combines the results. This allows the model to use context from both before AND after each word, improving understanding.\n", "\n", "## External tools & resources\n", "\n", "- **Stopwords**: Common words (e.g., \"the\", \"is\", \"at\") filtered out during preprocessing to reduce noise and focus on meaningful content.\n", " - *This project*: [NLTK Stopwords](https://www.nltk.org/nltk_data/) — curated lists for 20+ languages\n", " - *Alternatives*: [spaCy stopwords](https://spacy.io/usage/rule-based-matching#vocab-stopwords), [scikit-learn ENGLISH_STOP_WORDS](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.ENGLISH_STOP_WORDS.html)\n", "\n", "- **Tokenizers**: Tools that split text into tokens. Different tokenizers handle edge cases differently — social media text requires special handling for emoticons, hashtags, and informal spelling.\n", " - *This project*: [NLTK TweetTokenizer](https://www.nltk.org/api/nltk.tokenize.casual.html#nltk.tokenize.casual.TweetTokenizer) — handles emoticons, hashtags, mentions, and normalizes repeated characters (e.g., \"sooooo\" → \"sooo\")\n", " - *Alternatives*: [spaCy Tokenizer](https://spacy.io/usage/linguistic-features#tokenization), [Hugging Face Tokenizers](https://huggingface.co/docs/tokenizers/), [NLTK word_tokenize](https://www.nltk.org/api/nltk.tokenize.word_tokenize.html)\n", "\n", "- **Word Embeddings**: Pretrained vector representations that capture semantic relationships between words. Using pretrained embeddings transfers knowledge from large corpora to your model.\n", " - *This project*: [GloVe Twitter embeddings](https://nlp.stanford.edu/projects/glove/) — trained on 27B Twitter tokens, 100 dimensions\n", " - *Alternatives*: [Word2Vec](https://radimrehurek.com/gensim/models/word2vec.html), [FastText](https://fasttext.cc/docs/en/english-vectors.html), [Hugging Face sentence-transformers](https://huggingface.co/sentence-transformers)\n", "\n", "\n", "## Notebook setup\n", "\n", "### Imports" ] }, { "cell_type": "code", "execution_count": 1, "id": "2c49205e", "metadata": {}, "outputs": [], "source": [ "import os\n", "import re\n", "import zipfile\n", "import urllib.request\n", "from collections import Counter\n", "\n", "import nltk\n", "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "import tensorflow as tf\n", "from tensorflow.keras.models import Sequential\n", "from tensorflow.keras.layers import (\n", " Embedding, Dense, GRU, Dropout, Bidirectional, SpatialDropout1D\n", ")\n", "from tensorflow.keras.preprocessing.sequence import pad_sequences\n", "from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping, TensorBoard\n", "from tensorflow.keras.regularizers import l2\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.utils.class_weight import compute_class_weight\n", "from sklearn.metrics import confusion_matrix, roc_curve, auc, precision_recall_curve\n", "from sklearn.preprocessing import label_binarize" ] }, { "cell_type": "markdown", "id": "664d2ca9", "metadata": {}, "source": [ "### Configuration" ] }, { "cell_type": "code", "execution_count": 2, "id": "b8d56b31", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Using GPU: /physical_device:GPU:0\n" ] } ], "source": [ "# Configure GPU - use only GPU 0\n", "import os\n", "os.environ['CUDA_VISIBLE_DEVICES'] = '0'\n", "\n", "# Optionally limit GPU memory growth to avoid OOM errors\n", "gpus = tf.config.list_physical_devices('GPU')\n", "if gpus:\n", " try:\n", " tf.config.experimental.set_memory_growth(gpus[0], True)\n", " print(f'Using GPU: {gpus[0].name}')\n", " except RuntimeError as e:\n", " print(e)\n", "else:\n", " print('No GPU available, using CPU')\n", "\n", "# Download NLTK resources\n", "nltk.download('stopwords', quiet=True)\n", "from nltk.corpus import stopwords\n", "from nltk.tokenize import TweetTokenizer\n", "\n", "# Set up stop words and tokenizer\n", "stop_words = set(stopwords.words('english'))\n", "tweet_tokenizer = TweetTokenizer(preserve_case=False, reduce_len=True, strip_handles=True)" ] }, { "cell_type": "markdown", "id": "dc9b3cef", "metadata": {}, "source": [ "## 1. Data preparation\n", "\n", "### 1.1. Load" ] }, { "cell_type": "code", "execution_count": 3, "id": "08292db5", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idnamescoretext
0638060586258038784michael jackson005 Beat it - Michael Jackson - Thriller (25th ...
1638061181823922176michael jackson1Jay Z joins Instagram with nostalgic tribute t...
2638083821364244480michael jackson0Michael Jackson: Bad 25th Anniversary Edition ...
3638091450132078593michael jackson1I liked a @YouTube video http://t.co/AaR3pjp2P...
4638125563790557184michael jackson118th anniv of Princess Diana's death. I still ...
\n", "
" ], "text/plain": [ " id name score \\\n", "0 638060586258038784 michael jackson 0 \n", "1 638061181823922176 michael jackson 1 \n", "2 638083821364244480 michael jackson 0 \n", "3 638091450132078593 michael jackson 1 \n", "4 638125563790557184 michael jackson 1 \n", "\n", " text \n", "0 05 Beat it - Michael Jackson - Thriller (25th ... \n", "1 Jay Z joins Instagram with nostalgic tribute t... \n", "2 Michael Jackson: Bad 25th Anniversary Edition ... \n", "3 I liked a @YouTube video http://t.co/AaR3pjp2P... \n", "4 18th anniv of Princess Diana's death. I still ... " ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = pd.read_parquet('../data/twitter-2016.parquet')\n", "df.head()" ] }, { "cell_type": "code", "execution_count": 4, "id": "52be2521", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "RangeIndex: 30467 entries, 0 to 30466\n", "Data columns (total 4 columns):\n", " # Column Non-Null Count Dtype \n", "--- ------ -------------- ----- \n", " 0 id 30467 non-null int64 \n", " 1 name 30467 non-null object\n", " 2 score 30467 non-null int64 \n", " 3 text 30467 non-null object\n", "dtypes: int64(2), object(2)\n", "memory usage: 952.2+ KB\n" ] } ], "source": [ "df.info()" ] }, { "cell_type": "markdown", "id": "4b4c04aa", "metadata": {}, "source": [ "### 1.2. Clean" ] }, { "cell_type": "code", "execution_count": 5, "id": "bf67ece0", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Cleaned dataset: 30,467 rows (0 dropped)\n", "Score distribution: {0: 299, 1: 3392, 2: 12953, 3: 12812, 4: 1011}\n" ] } ], "source": [ "# Clean data - drop rows with missing scores\n", "df_clean = df.dropna(subset=['score']).copy()\n", "df_clean['score'] = df_clean['score'].astype(int)\n", "\n", "# Shift scores to 0-based index: [-2,-1,0,1,2] -> [0,1,2,3,4]\n", "score_min = df_clean['score'].min()\n", "df_clean['score_shifted'] = df_clean['score'] - score_min\n", "\n", "print(f'Cleaned dataset: {len(df_clean):,} rows ({len(df) - len(df_clean):,} dropped)')\n", "print(f'Score distribution: {df_clean[\"score_shifted\"].value_counts().sort_index().to_dict()}')" ] }, { "cell_type": "markdown", "id": "7b45c1eb", "metadata": {}, "source": [ "### 1.3. Tokenize" ] }, { "cell_type": "code", "execution_count": 6, "id": "2b7efdeb", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
texttokensscore
005 Beat it - Michael Jackson - Thriller (25th ...[05, beat, michael, jackson, thriller, 25th, a...0
1Jay Z joins Instagram with nostalgic tribute t...[jay, joins, instagram, nostalgic, tribute, mi...1
2Michael Jackson: Bad 25th Anniversary Edition ...[michael, jackson, bad, 25th, anniversary, edi...0
3I liked a @YouTube video http://t.co/AaR3pjp2P...[liked, video, one, direction, singing, man, m...1
418th anniv of Princess Diana's death. I still ...[18th, anniv, princess, diana's, death, still,...1
\n", "
" ], "text/plain": [ " text \\\n", "0 05 Beat it - Michael Jackson - Thriller (25th ... \n", "1 Jay Z joins Instagram with nostalgic tribute t... \n", "2 Michael Jackson: Bad 25th Anniversary Edition ... \n", "3 I liked a @YouTube video http://t.co/AaR3pjp2P... \n", "4 18th anniv of Princess Diana's death. I still ... \n", "\n", " tokens score \n", "0 [05, beat, michael, jackson, thriller, 25th, a... 0 \n", "1 [jay, joins, instagram, nostalgic, tribute, mi... 1 \n", "2 [michael, jackson, bad, 25th, anniversary, edi... 0 \n", "3 [liked, video, one, direction, singing, man, m... 1 \n", "4 [18th, anniv, princess, diana's, death, still,... 1 " ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Clean and tokenize text using NLTK's TweetTokenizer\n", "def clean_text(text):\n", " '''Clean and tokenize tweet text using TweetTokenizer.'''\n", "\n", " if not isinstance(text, str):\n", " return []\n", "\n", " # Remove URLs before tokenizing\n", " text = re.sub(r'http\\S+|www\\S+|https\\S+', '', text)\n", "\n", " # TweetTokenizer handles: lowercase, @mentions, repeated chars (e.g., sooooo -> sooo)\n", " tokens = tweet_tokenizer.tokenize(text)\n", "\n", " # Remove stopwords and single characters\n", " tokens = [t for t in tokens if t not in stop_words and len(t) > 1]\n", "\n", " return tokens\n", "\n", "# Apply tokenization\n", "df_clean['tokens'] = df_clean['text'].apply(clean_text)\n", "df_clean[['text', 'tokens', 'score']].head()" ] }, { "cell_type": "markdown", "id": "9a70b43c", "metadata": {}, "source": [ "### 1.4. Build vocabulary" ] }, { "cell_type": "code", "execution_count": 7, "id": "0d8efd85", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Vocabulary size: 14965\n" ] } ], "source": [ "# Build vocabulary\n", "def build_vocab(token_lists, min_freq=2):\n", " '''Build vocabulary from token lists.'''\n", "\n", " counter = Counter()\n", "\n", " for tokens in token_lists:\n", " counter.update(tokens)\n", " \n", " # Filter by minimum frequency and create vocab\n", " vocab = {'': 0, '': 1}\n", "\n", " for word, freq in counter.items():\n", " if freq >= min_freq:\n", " vocab[word] = len(vocab)\n", "\n", " return vocab\n", "\n", "vocab = build_vocab(df_clean['tokens'])\n", "print(f'Vocabulary size: {len(vocab)}')" ] }, { "cell_type": "markdown", "id": "cc6059bd", "metadata": {}, "source": [ "### 1.5. Convert to indices" ] }, { "cell_type": "code", "execution_count": 8, "id": "951ab5c1", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "X shape: (30467, 50)\n", "y shape: (30467,)\n" ] } ], "source": [ "def tokens_to_indices(tokens_list, vocab, max_len=50):\n", " '''Convert token lists to padded index sequences.'''\n", "\n", " sequences = []\n", "\n", " for tokens in tokens_list:\n", " indices = [vocab.get(token, vocab['']) for token in tokens]\n", " sequences.append(indices)\n", "\n", " # Pad sequences to max_len\n", " return pad_sequences(sequences, maxlen=max_len, padding='post', value=vocab[''])\n", "\n", "# Convert all data to indices\n", "max_len = 50\n", "X = tokens_to_indices(df_clean['tokens'].tolist(), vocab, max_len)\n", "y = df_clean['score_shifted'].values\n", "\n", "print(f'X shape: {X.shape}')\n", "print(f'y shape: {y.shape}')" ] }, { "cell_type": "markdown", "id": "5118247a", "metadata": {}, "source": [ "### 1.6. Train/test split" ] }, { "cell_type": "code", "execution_count": 9, "id": "c941eef5", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "X_train shape: (24373, 50)\n", "X_test shape: (6094, 50)\n", "Score distribution (train): Counter({2: 10362, 3: 10249, 1: 2714, 4: 809, 0: 239})\n" ] } ], "source": [ "# Train/test split\n", "X_train, X_test, y_train, y_test = train_test_split(\n", " X, y,\n", " test_size=0.2,\n", " random_state=42,\n", " stratify=y\n", ")\n", "\n", "print(f'X_train shape: {X_train.shape}')\n", "print(f'X_test shape: {X_test.shape}')\n", "print(f'Score distribution (train): {Counter(y_train)}')" ] }, { "cell_type": "markdown", "id": "45f38bea", "metadata": {}, "source": [ "## 2. Prepare GloVe embeddings\n", "\n", "GloVe (Global Vectors) provides pretrained word embeddings trained on large text corpora. Using these gives our model a head start - words already have meaningful representations instead of random vectors.\n", "\n", "### 2.1. Download embeddings" ] }, { "cell_type": "code", "execution_count": 10, "id": "74bb2bfa", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "GloVe file already exists: ../data/glove.twitter.27B.100d.txt\n" ] } ], "source": [ "# Download GloVe embeddings (Twitter-specific, 100d)\n", "glove_dir = '../data'\n", "glove_file = os.path.join(glove_dir, 'glove.twitter.27B.100d.txt')\n", "glove_zip = os.path.join(glove_dir, 'glove.twitter.27B.zip')\n", "glove_url = 'https://nlp.stanford.edu/data/glove.twitter.27B.zip'\n", "\n", "if not os.path.exists(glove_file):\n", "\n", " print('Downloading GloVe embeddings...')\n", " urllib.request.urlretrieve(glove_url, glove_zip)\n", "\n", " print('Extracting...')\n", " with zipfile.ZipFile(glove_zip, 'r') as zip_ref:\n", " zip_ref.extract('glove.twitter.27B.100d.txt', glove_dir)\n", "\n", " os.remove(glove_zip)\n", " print('Done!')\n", "\n", "else:\n", " print(f'GloVe file already exists: {glove_file}')" ] }, { "cell_type": "markdown", "id": "61a52fea", "metadata": {}, "source": [ "### 2.2. Load embeddings into dictionary" ] }, { "cell_type": "code", "execution_count": 11, "id": "5cf57e38", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Loading GloVe vectors...\n", "Loaded 1,193,514 word vectors\n", "Embedding dimension: 100\n" ] } ], "source": [ "# Load GloVe vectors into dictionary\n", "print('Loading GloVe vectors...')\n", "\n", "glove_vectors = {}\n", "\n", "with open(glove_file, 'r', encoding='utf-8') as f:\n", " for line in f:\n", "\n", " values = line.split()\n", " word = values[0]\n", " vector = np.array(values[1:], dtype='float32')\n", " glove_vectors[word] = vector\n", "\n", "print(f'Loaded {len(glove_vectors):,} word vectors')\n", "print(f'Embedding dimension: {len(next(iter(glove_vectors.values())))}')" ] }, { "cell_type": "markdown", "id": "229ff779", "metadata": {}, "source": [ "### 2.3. Create embedding look-up table for our vocabulary" ] }, { "cell_type": "code", "execution_count": 12, "id": "d1cc20ab", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Vocabulary coverage: 12,976 / 14,965 (86.7%)\n", "Embedding matrix shape: (14965, 100)\n" ] } ], "source": [ "# Create embedding matrix for our vocabulary\n", "embedding_dim = 100 # GloVe Twitter uses 100d\n", "vocab_size = len(vocab)\n", "\n", "embedding_matrix = np.zeros((vocab_size, embedding_dim))\n", "words_found = 0\n", "\n", "for word, idx in vocab.items():\n", " if word in glove_vectors:\n", " embedding_matrix[idx] = glove_vectors[word]\n", " words_found += 1\n", "\n", " else:\n", " # Random initialization for words not in GloVe\n", " embedding_matrix[idx] = np.random.normal(scale=0.6, size=(embedding_dim,))\n", "\n", "coverage = words_found / vocab_size\n", "print(f'Vocabulary coverage: {words_found:,} / {vocab_size:,} ({coverage:.1%})')\n", "print(f'Embedding matrix shape: {embedding_matrix.shape}')" ] }, { "cell_type": "markdown", "id": "e889cccf", "metadata": {}, "source": [ "## 3. Baseline: Bidirectional GRU model\n", "\n", "First, let's establish our baseline performance with a pure recurrent architecture.\n", "\n", "### 3.1. Build model" ] }, { "cell_type": "code", "execution_count": 13, "id": "d90c1cf5", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:Layer gru will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n", "WARNING:tensorflow:Layer gru will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n", "WARNING:tensorflow:Layer gru will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n", "WARNING:tensorflow:Layer gru_1 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n", "WARNING:tensorflow:Layer gru_1 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n", "WARNING:tensorflow:Layer gru_1 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n", "Model: \"sequential\"\n", "_________________________________________________________________\n", " Layer (type) Output Shape Param # \n", "=================================================================\n", " embedding (Embedding) (None, None, 100) 1496500 \n", " \n", " spatial_dropout1d (Spatial (None, None, 100) 0 \n", " Dropout1D) \n", " \n", " bidirectional (Bidirection (None, None, 128) 63744 \n", " al) \n", " \n", " bidirectional_1 (Bidirecti (None, 64) 31104 \n", " onal) \n", " \n", " dropout (Dropout) (None, 64) 0 \n", " \n", " dense (Dense) (None, 5) 325 \n", " \n", "=================================================================\n", "Total params: 1591673 (6.07 MB)\n", "Trainable params: 1591673 (6.07 MB)\n", "Non-trainable params: 0 (0.00 Byte)\n", "_________________________________________________________________\n" ] } ], "source": [ "# Model hyperparameters\n", "hidden_dim = 64\n", "output_dim = len(set(y_train))\n", "dropout = 0.3 # Spatial dropout rate\n", "recurrent_dropout = 0.3 # Dropout on recurrent connections\n", "l2_reg = 0.01 # L2 regularization strength\n", "learning_rate = 0.0001 # Lower LR for fine-tuning embeddings\n", "\n", "# Build Bidirectional GRU model with pretrained GloVe embeddings\n", "model = Sequential([\n", " Embedding(\n", " vocab_size, \n", " embedding_dim,\n", " weights=[embedding_matrix],\n", " trainable=True # Fine-tune embeddings with lower learning rate\n", " ),\n", " SpatialDropout1D(dropout),\n", " Bidirectional(GRU(hidden_dim, recurrent_dropout=recurrent_dropout, return_sequences=True)),\n", " Bidirectional(GRU(hidden_dim // 2, recurrent_dropout=recurrent_dropout)), # Second layer, smaller\n", " Dropout(dropout),\n", " Dense(output_dim, activation='softmax', kernel_regularizer=l2(l2_reg))\n", "])\n", "\n", "model.compile(\n", " optimizer=tf.keras.optimizers.Adam(learning_rate=learning_rate),\n", " loss='sparse_categorical_crossentropy',\n", " metrics=['accuracy']\n", ")\n", "\n", "model.summary()" ] }, { "cell_type": "markdown", "id": "1f4d6bcf", "metadata": {}, "source": [ "### 3.2. Define training callbacks" ] }, { "cell_type": "code", "execution_count": 14, "id": "9e1fb763", "metadata": {}, "outputs": [], "source": [ "callbacks = [\n", " ModelCheckpoint(\n", " '../models/rnn_classifier.keras',\n", " save_best_only=True,\n", " monitor='val_accuracy',\n", " verbose=1\n", " ),\n", " EarlyStopping(\n", " monitor='val_loss',\n", " patience=5,\n", " restore_best_weights=True,\n", " verbose=1\n", " ),\n", " TensorBoard(\n", " log_dir='../logs/rnn_classifier',\n", " histogram_freq=1,\n", " write_graph=True\n", " )\n", "]" ] }, { "cell_type": "markdown", "id": "d9cae247", "metadata": {}, "source": [ "### 3.3. Calculate class weights" ] }, { "cell_type": "code", "execution_count": 15, "id": "f10adb3a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Class weights: {0: 20.39581589958159, 1: 1.7960943257184967, 2: 0.47043041883806214, 3: 0.47561713337886624, 4: 6.025463535228678}\n" ] } ], "source": [ "class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)\n", "class_weight_dict = dict(enumerate(class_weights))\n", "print(f'Class weights: {class_weight_dict}')" ] }, { "cell_type": "markdown", "id": "0791aaeb", "metadata": {}, "source": [ "### 3.4. Train the model" ] }, { "cell_type": "code", "execution_count": null, "id": "9fb97944", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1/30\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n", "I0000 00:00:1771291113.224545 1145675 service.cc:145] XLA service 0x7284248102b0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:\n", "I0000 00:00:1771291113.224592 1145675 service.cc:153] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0\n", "I0000 00:00:1771291113.333135 1145675 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "381/381 [==============================] - ETA: 0s - loss: 1.7533 - accuracy: 0.2112\n", "Epoch 1: val_accuracy improved from -inf to 0.24467, saving model to ../models/rnn_classifier.keras\n", "381/381 [==============================] - 347s 785ms/step - loss: 1.7533 - accuracy: 0.2112 - val_loss: 1.6363 - val_accuracy: 0.2447\n", "Epoch 2/30\n", "381/381 [==============================] - ETA: 0s - loss: 1.6478 - accuracy: 0.2403\n", "Epoch 2: val_accuracy improved from 0.24467 to 0.32294, saving model to ../models/rnn_classifier.keras\n", "381/381 [==============================] - 283s 744ms/step - loss: 1.6478 - accuracy: 0.2403 - val_loss: 1.5840 - val_accuracy: 0.3229\n", "Epoch 3/30\n", "381/381 [==============================] - ETA: 0s - loss: 1.5553 - accuracy: 0.2540\n", "Epoch 3: val_accuracy did not improve from 0.32294\n", "381/381 [==============================] - 284s 746ms/step - loss: 1.5553 - accuracy: 0.2540 - val_loss: 1.4960 - val_accuracy: 0.3118\n", "Epoch 4/30\n", "381/381 [==============================] - ETA: 0s - loss: 1.4812 - accuracy: 0.2677\n", "Epoch 4: val_accuracy improved from 0.32294 to 0.33640, saving model to ../models/rnn_classifier.keras\n", "381/381 [==============================] - 284s 745ms/step - loss: 1.4812 - accuracy: 0.2677 - val_loss: 1.4441 - val_accuracy: 0.3364\n", "Epoch 5/30\n", "247/381 [==================>...........] - ETA: 1:36 - loss: 1.4612 - accuracy: 0.2767" ] } ], "source": [ "# Training with class weighting and early stopping\n", "history = model.fit(\n", " X_train, y_train,\n", " validation_data=(X_test, y_test),\n", " epochs=30,\n", " batch_size=64,\n", " class_weight=class_weight_dict,\n", " callbacks=callbacks\n", ")" ] }, { "cell_type": "markdown", "id": "72de8d3b", "metadata": {}, "source": [ "## 4. Baseline evaluation\n", "\n", "Record the baseline metrics below so you can compare them to your CNN-RNN model later.\n", "\n", "### 4.1. Make test set predictions" ] }, { "cell_type": "code", "execution_count": null, "id": "7342f545", "metadata": {}, "outputs": [], "source": [ "# Load best model and get predictions\n", "model = tf.keras.models.load_model('../models/rnn_classifier.keras')\n", "\n", "# Get predicted probabilities and classes\n", "y_prob = model.predict(X_test)\n", "y_pred = y_prob.argmax(axis=1)\n", "\n", "# Class labels\n", "class_names = ['-2', '-1', '0', '1', '2']\n", "n_classes = len(class_names)\n", "\n", "print(f'Test samples: {len(y_test)}')\n", "print(f'Prediction distribution:')\n", "print(f' Actual: {Counter(y_test)}')\n", "print(f' Predicted: {Counter(y_pred)}')" ] }, { "cell_type": "markdown", "id": "408d5b20", "metadata": {}, "source": [ "### 4.2. Per-class accuracy" ] }, { "cell_type": "code", "execution_count": null, "id": "89df382c", "metadata": {}, "outputs": [], "source": [ "# Per-class accuracy table\n", "per_class_stats = []\n", "\n", "for i, class_name in enumerate(class_names):\n", " mask = y_test == i\n", " n_samples = mask.sum()\n", " n_correct = (y_pred[mask] == i).sum()\n", " accuracy = n_correct / n_samples if n_samples > 0 else 0\n", " \n", " per_class_stats.append({\n", " 'Class': class_name,\n", " 'Samples': n_samples,\n", " 'Correct': n_correct,\n", " 'Accuracy': f'{accuracy:.1%}'\n", " })\n", "\n", "# Add overall accuracy\n", "overall_acc = (y_pred == y_test).mean()\n", "per_class_stats.append({\n", " 'Class': 'Overall',\n", " 'Samples': len(y_test),\n", " 'Correct': (y_pred == y_test).sum(),\n", " 'Accuracy': f'{overall_acc:.1%}'\n", "})\n", "\n", "accuracy_df = pd.DataFrame(per_class_stats)\n", "accuracy_df" ] }, { "cell_type": "markdown", "id": "296a3976", "metadata": {}, "source": [ "### 4.3. Confusion matrix" ] }, { "cell_type": "code", "execution_count": null, "id": "02d44a72", "metadata": {}, "outputs": [], "source": [ "# Plot confusion matrix\n", "fig, ax = plt.subplots(figsize=(8, 6))\n", "\n", "ax.set_title('Confusion matrix')\n", "\n", "cm = confusion_matrix(y_test, y_pred)\n", "\n", "sns.heatmap(\n", " cm, \n", " annot=True, fmt='d', cmap='Blues', \n", " xticklabels=class_names, yticklabels=class_names, ax=ax\n", ")\n", "\n", "ax.set_xlabel('Predicted')\n", "ax.set_ylabel('Actual')\n", "\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "7895bdbc", "metadata": {}, "source": [ "### 4.4. Predicted probability distributions by class" ] }, { "cell_type": "code", "execution_count": null, "id": "91f28172", "metadata": {}, "outputs": [], "source": [ "# Predicted probability distributions for each true class\n", "fig, axes = plt.subplots(1, n_classes, figsize=(10, 2.5))\n", "\n", "plt.suptitle('Predicted probability distributions by true class', y=1.02)\n", "\n", "for i, class_name in enumerate(class_names):\n", "\n", " # Get predictions for samples of this true class\n", " mask = y_test == i\n", " probs_for_class = y_prob[mask]\n", " \n", " # Plot distribution of predicted probabilities for each predicted class\n", " ax = axes[i]\n", " ax.boxplot([probs_for_class[:, j] for j in range(n_classes)], tick_labels=class_names)\n", " ax.set_title(f'True class: {class_name}')\n", " ax.set_xlabel('Predicted class')\n", " ax.set_ylabel('Probability')\n", " ax.set_ylim(0, 1)\n", "\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "9fa8cfff", "metadata": {}, "source": [ "### 4.5. Evaluation curves" ] }, { "cell_type": "code", "execution_count": null, "id": "5a0b7d21", "metadata": {}, "outputs": [], "source": [ "# Binarize labels for multiclass ROC/PR\n", "y_test_bin = label_binarize(y_test, classes=range(n_classes))\n", "\n", "# Compute ROC and PR curves for each class\n", "fig, axes = plt.subplots(1, 2, figsize=(10, 4))\n", "\n", "# ROC curves\n", "ax_roc = axes[0]\n", "\n", "for i, class_name in enumerate(class_names):\n", " fpr, tpr, _ = roc_curve(y_test_bin[:, i], y_prob[:, i])\n", " roc_auc = auc(fpr, tpr)\n", " ax_roc.plot(fpr, tpr, label=f'Class {class_name} (AUC={roc_auc:.2f})')\n", "\n", "ax_roc.plot([0, 1], [0, 1], 'k--', alpha=0.5)\n", "ax_roc.set_xlabel('False positive rate')\n", "ax_roc.set_ylabel('True positive rate')\n", "ax_roc.set_title('ROC curves (one-vs-rest)')\n", "ax_roc.legend(loc='lower right')\n", "ax_roc.set_xlim([0, 1])\n", "ax_roc.set_ylim([0, 1.05])\n", "\n", "# PR curves\n", "ax_pr = axes[1]\n", "\n", "for i, class_name in enumerate(class_names):\n", " precision, recall, _ = precision_recall_curve(y_test_bin[:, i], y_prob[:, i])\n", " pr_auc = auc(recall, precision)\n", " ax_pr.plot(recall, precision, label=f'Class {class_name} (AUC={pr_auc:.2f})')\n", "\n", "ax_pr.set_xlabel('Recall')\n", "ax_pr.set_ylabel('Precision')\n", "ax_pr.set_title('Precision-recall curves (one-vs-rest)')\n", "ax_pr.legend(loc='lower left')\n", "ax_pr.set_xlim([0, 1])\n", "ax_pr.set_ylim([0, 1.05])\n", "\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "a59b2536", "metadata": {}, "source": [ "---\n", "\n", "## 5. Activity: Build a CNN-RNN hybrid model (SOLUTION)\n", "\n", "The solution below demonstrates one effective approach: adding a Conv1D layer before the GRU to capture local n-gram patterns.\n", "\n", "### Why add convolutions?\n", "\n", "- **Local pattern detection**: Conv1D layers can detect n-gram patterns (e.g., \"not good\", \"very happy\") that are important for sentiment\n", "- **Dimensionality reduction**: Convolutions can reduce sequence length before the RNN, making training faster\n", "- **Feature extraction**: The CNN extracts local features, then the RNN captures long-range dependencies\n", "\n", "### Solution architecture\n", "\n", "```text\n", "Embedding (vocab_size × 100)\n", " ↓\n", "SpatialDropout1D (0.3)\n", " ↓\n", "Conv1D (128 filters, kernel_size=3, relu, padding='same') ← Captures trigram patterns\n", " ↓\n", "MaxPooling1D (pool_size=2) ← Reduces sequence length by half\n", " ↓\n", "Bidirectional GRU (32 units, return_sequences=True)\n", " ↓\n", "Bidirectional GRU (16 units)\n", " ↓\n", "Dropout (0.3)\n", " ↓\n", "Dense (5 classes, softmax)\n", "```\n", "\n", "### Key design decisions\n", "\n", "1. **128 filters**: Each filter learns a different pattern detector. More filters = more patterns, but more parameters.\n", "\n", "2. **kernel_size=3**: Captures trigrams like \"not very good\". You could try 4 or 5 for longer patterns.\n", "\n", "3. **padding='same'**: Output length = input length, so MaxPooling works correctly.\n", "\n", "4. **MaxPooling1D(pool_size=2)**: Halves the sequence length, making the GRU faster and helping it focus on the most salient features.\n", "\n", "5. **Smaller GRU layers**: Since the Conv1D already extracts features, we can use smaller GRU layers (32/16 instead of 64/32)." ] }, { "cell_type": "code", "execution_count": null, "id": "ff1c7488", "metadata": {}, "outputs": [], "source": [ "from tensorflow.keras.layers import Conv1D, MaxPooling1D\n", "\n", "cnn_rnn_model = Sequential([\n", " Embedding(\n", " vocab_size, \n", " embedding_dim,\n", " weights=[embedding_matrix],\n", " trainable=True\n", " ),\n", " SpatialDropout1D(dropout),\n", " Conv1D(filters=128, kernel_size=3, activation='relu', padding='same'),\n", " MaxPooling1D(pool_size=2),\n", " Bidirectional(GRU(hidden_dim // 2, recurrent_dropout=recurrent_dropout, return_sequences=True)),\n", " Bidirectional(GRU(hidden_dim // 4, recurrent_dropout=recurrent_dropout)),\n", " Dropout(dropout),\n", " Dense(output_dim, activation='softmax', kernel_regularizer=l2(l2_reg))\n", "])\n", "\n", "cnn_rnn_model.compile(\n", " optimizer=tf.keras.optimizers.Adam(learning_rate=learning_rate),\n", " loss='sparse_categorical_crossentropy',\n", " metrics=['accuracy']\n", ")\n", "\n", "cnn_rnn_model.summary()" ] }, { "cell_type": "markdown", "id": "d4dfef75", "metadata": {}, "source": [ "### 5.1. Train your CNN-RNN model" ] }, { "cell_type": "code", "execution_count": null, "id": "35d2245e", "metadata": {}, "outputs": [], "source": [ "# Define callbacks for CNN-RNN model (saves to different file)\n", "cnn_rnn_callbacks = [\n", " ModelCheckpoint(\n", " '../models/cnn_rnn_classifier.keras',\n", " save_best_only=True,\n", " monitor='val_accuracy',\n", " verbose=1\n", " ),\n", " EarlyStopping(\n", " monitor='val_loss',\n", " patience=5,\n", " restore_best_weights=True,\n", " verbose=1\n", " )\n", "]\n", "\n", "# Train the CNN-RNN model\n", "cnn_rnn_history = cnn_rnn_model.fit(\n", " X_train, y_train,\n", " validation_data=(X_test, y_test),\n", " epochs=30,\n", " batch_size=64,\n", " class_weight=class_weight_dict,\n", " callbacks=cnn_rnn_callbacks\n", ")" ] }, { "cell_type": "markdown", "id": "7951f272", "metadata": {}, "source": [ "### 5.2. Evaluate your CNN-RNN model" ] }, { "cell_type": "code", "execution_count": null, "id": "3bc6b841", "metadata": {}, "outputs": [], "source": [ "# Load best CNN-RNN model and get predictions\n", "cnn_rnn_model = tf.keras.models.load_model('../models/cnn_rnn_classifier.keras')\n", "\n", "# Get predicted probabilities and classes\n", "y_prob_cnn = cnn_rnn_model.predict(X_test)\n", "y_pred_cnn = y_prob_cnn.argmax(axis=1)\n", "\n", "# Calculate accuracy\n", "cnn_rnn_accuracy = (y_pred_cnn == y_test).mean()\n", "print(f'CNN-RNN model accuracy: {cnn_rnn_accuracy:.1%}')" ] }, { "cell_type": "markdown", "id": "5dee91fe", "metadata": {}, "source": [ "---\n", "\n", "## 6. Compare models\n", "\n", "Now let's compare your CNN-RNN model to the baseline GRU model." ] }, { "cell_type": "code", "execution_count": null, "id": "03720dce", "metadata": {}, "outputs": [], "source": [ "# Load both models\n", "baseline_model = tf.keras.models.load_model('../models/rnn_classifier.keras')\n", "cnn_rnn_model = tf.keras.models.load_model('../models/cnn_rnn_classifier.keras')\n", "\n", "# Get predictions from both models\n", "y_pred_baseline = baseline_model.predict(X_test).argmax(axis=1)\n", "y_pred_cnn_rnn = cnn_rnn_model.predict(X_test).argmax(axis=1)\n", "\n", "# Calculate overall accuracy\n", "baseline_acc = (y_pred_baseline == y_test).mean()\n", "cnn_rnn_acc = (y_pred_cnn_rnn == y_test).mean()\n", "\n", "# Create comparison dataframe\n", "comparison = pd.DataFrame({\n", " 'Model': ['Baseline GRU', 'CNN-RNN'],\n", " 'Accuracy': [f'{baseline_acc:.1%}', f'{cnn_rnn_acc:.1%}'],\n", " 'Parameters': [baseline_model.count_params(), cnn_rnn_model.count_params()]\n", "})\n", "\n", "print('Model comparison:')\n", "comparison" ] }, { "cell_type": "code", "execution_count": null, "id": "53cf2d4d", "metadata": {}, "outputs": [], "source": [ "# Side-by-side confusion matrices\n", "fig, axes = plt.subplots(1, 2, figsize=(14, 5))\n", "\n", "for ax, y_pred, title in zip(\n", " axes, \n", " [y_pred_baseline, y_pred_cnn_rnn], \n", " ['Baseline GRU', 'CNN-RNN']\n", "):\n", " cm = confusion_matrix(y_test, y_pred)\n", " sns.heatmap(\n", " cm, annot=True, fmt='d', cmap='Blues',\n", " xticklabels=class_names, yticklabels=class_names, ax=ax\n", " )\n", " ax.set_title(f'{title} (Acc: {(y_pred == y_test).mean():.1%})')\n", " ax.set_xlabel('Predicted')\n", " ax.set_ylabel('Actual')\n", "\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "6d87f777", "metadata": {}, "source": [ "## 7. Reflection\n", "\n", "Answer the following questions:\n", "\n", "1. **Did your CNN-RNN model outperform the baseline?** Why or why not?\n", "\n", "2. **How did the number of parameters change?** Is there a trade-off between model complexity and performance?\n", "\n", "3. **Which classes improved the most?** Look at the per-class accuracy changes.\n", "\n", "4. **What other architectures could you try?** Consider:\n", " - Multiple conv layers with different kernel sizes\n", " - Using only CNN (no RNN) with GlobalMaxPooling\n", " - LSTM instead of GRU\n", " - Attention mechanisms" ] }, { "cell_type": "markdown", "id": "0d23f8d6", "metadata": {}, "source": [ "### Solution notes\n", "\n", "**1. Did your CNN-RNN model outperform the baseline?**\n", "\n", "Results vary by run, but typically the CNN-RNN model shows marginal improvement or similar performance. The convolutional layer helps by:\n", "- Detecting local sentiment indicators like \"not good\" or \"very happy\" as fixed patterns\n", "- Reducing sequence length before the RNN, which can help with longer sequences\n", "- Extracting position-invariant features (the same pattern is detected anywhere in the tweet)\n", "\n", "However, for short tweets (average ~10-15 tokens), the baseline GRU is already quite capable of capturing the full context, limiting the CNN's advantage.\n", "\n", "**2. How did the number of parameters change?**\n", "\n", "The CNN-RNN model typically has **more parameters** due to the Conv1D layer (128 filters × 3 kernel × 100 embedding_dim = 38,400 weights + biases). However, we reduced the GRU sizes to partially compensate.\n", "\n", "Trade-off: More parameters = more capacity to learn complex patterns, but also higher risk of overfitting on small datasets. With ~15K training samples, regularization (dropout, L2) is important.\n", "\n", "**3. Which classes improved the most?**\n", "\n", "Typically, the extreme sentiment classes (-2 and +2) and the neutral class (0) see the most variability:\n", "- **Extreme classes**: Often underrepresented, so small changes in the model can have outsized effects\n", "- **Neutral class**: Hard to distinguish from mild sentiment; n-gram patterns like \"it's okay\" or \"not bad\" can help\n", "\n", "The adjacent classes (-1, +1) are hardest to improve because they're semantically closest to neutral.\n", "\n", "**4. What other architectures could you try?**\n", "\n", "- **Multiple kernel sizes**: Use parallel Conv1D layers with kernel_size=2,3,4,5 to capture different n-gram lengths, then concatenate (requires Functional API)\n", "- **Deeper convolutions**: Stack multiple Conv1D layers with increasing filters (64 → 128 → 256)\n", "- **Pure CNN**: Replace GRU entirely with GlobalMaxPooling1D — often competitive and much faster to train\n", "- **Attention mechanisms**: Add self-attention after the RNN to let the model focus on important words\n", "- **Transformer-based**: Use a pretrained model like BERT or DistilBERT for state-of-the-art results" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.12" } }, "nbformat": 4, "nbformat_minor": 5 }