Question 2: RNN for Shakespeare Text Generation

Objective

Train a Recurrent Neural Network (RNN) on Shakespeare text for next-word prediction. The implementation uses custom word embeddings and generates coherent text sequences by predicting the next word in a sequence.

Requirements

Software Dependencies

torch>=1.9.0
numpy>=1.21.0
matplotlib>=3.5.0
re (built-in)
collections (built-in)
math (built-in)

Hardware Requirements

Minimum: CPU with 4GB RAM
Recommended: GPU with 2GB+ VRAM (optional but faster)
Storage: ~50MB for models and outputs

Installation

Install required packages:

pip install torch numpy matplotlib

Verify installation:

import torch
import numpy as np
import matplotlib.pyplot as plt
print("All dependencies installed successfully!")

File Structure

Question2_RNN/
│
├── rnn_shakespeare_clean.py   # Main implementation file
├── README.md                  # This file
├── requirements.txt           # Package dependencies
└── outputs/                   # Generated outputs (created automatically)
    ├── plots/                 # Training curves and visualizations
    ├── models/                # Saved model checkpoints
    ├── generated_text/        # Sample generated texts
    └── results/               # Performance metrics and tables

Usage

Basic Usage

python rnn_shakespeare_clean.py

Advanced Usage with Custom Parameters

from rnn_shakespeare_clean import *

# Load data
text_data = load_shakespeare_data()
vocab = build_vocabulary(text_data)

# Create custom model
model = RNNModel(
    vocab_size=len(vocab),
    embedding_dim=256,     # Embedding dimension
    hidden_size=512,       # Hidden state size
    num_layers=3,          # Number of RNN layers
    rnn_type='LSTM'        # RNN type: 'RNN', 'LSTM', 'GRU'
)

# Generate custom text
generated = generate_text(model, dataset, "To be or not to", max_length=30, temperature=0.8)

Expected Outputs

1. Data Loading and Preprocessing

RNN for Shakespeare Text Generation
==================================================
Loading Shakespeare dataset...
Loaded 10 Shakespeare passages
Total characters: 8,247
Sample text: To be, or not to be, that is the question:
Whether 'tis nobler in the mind to suffer...
Building vocabulary...
Vocabulary size: 1,234
Most common words: [('the', 156), ('and', 134), ('to', 98)]

2. Training Progress

Training RNN for 10 epochs...
Epoch [1/10], Step [0/25], Loss: 6.2341
Epoch [1/10], Step [10/25], Loss: 5.8934
Epoch [1/10] - Average Loss: 5.4567
Epoch [1/10] - Perplexity: 234.12

Epoch [10/10] - Average Loss: 2.1456
Epoch [10/10] - Perplexity: 8.54

3. Text Generation Examples

TEXT GENERATION EXAMPLES
==================================================
Seed: 'To be or not to'
Generated: 'To be or not to be that is the question of life and death'

Seed: 'Romeo Romeo wherefore'
Generated: 'Romeo Romeo wherefore art thou romeo deny thy father and refuse thy name'

Seed: 'All the world's a'
Generated: 'All the world's a stage and all the men and women merely players'

4. Ablation Study Results

RNN ABLATION STUDY RESULTS
============================================================
Component       Value      Val Loss   Perplexity
------------------------------------------------------------
Hidden Size     128        3.4567     31.56     
Hidden Size     256        2.8934     18.12     
Hidden Size     512        2.7123     15.06     
RNN Type        RNN        4.1234     61.78     
RNN Type        LSTM       2.8934     18.12     
RNN Type        GRU        2.9456     19.01

5. Final Performance Metrics

FINAL MODEL PERFORMANCE
==================================================
Validation Loss: 2.8934
Validation Perplexity: 18.12
Vocabulary Size: 1,234

Implementation Details

RNN Architecture

RNNModel(
  (embedding): Embedding(1234, 128)     # Word embeddings
  (rnn): LSTM(128, 256, num_layers=2, batch_first=True, dropout=0.5)
  (dropout): Dropout(p=0.5)
  (fc): Linear(in_features=256, out_features=1234)
)

Key Components

1. Text Processing Pipeline

Tokenization: Splits text into words, handles punctuation
Vocabulary Building: Creates word-to-index mappings
Sequence Creation: Generates input-target pairs for training

2. Custom Word Embeddings

Trainable Embeddings: Learn word representations during training
Embedding Dimension: Configurable (default: 128)
No Pre-trained: Uses randomly initialized embeddings

3. RNN Variants Supported

Vanilla RNN: Simple recurrent connections
LSTM: Long Short-Term Memory (recommended)
GRU: Gated Recurrent Unit (faster alternative)

4. Text Generation Strategy

Autoregressive: Generates one word at a time
Temperature Sampling: Controls randomness in generation
Seed-based: Starts generation from provided seed phrase

Shakespeare Text Collection

Built-in passages include:

Hamlet's "To be or not to be" soliloquy
Romeo and Juliet balcony scene
"All the world's a stage" from As You Like It
Macbeth's "Tomorrow and tomorrow" speech
Julius Caesar's "Friends, Romans, countrymen"
Henry V's "Once more unto the breach"
And more famous passages

Results Interpretation

Training Metrics

Loss Progression

Initial Loss: 6.0+ (random predictions)
Mid Training: 3.0-4.0 (learning patterns)
Final Loss: 2.0-3.0 (good text modeling)

Perplexity Analysis

# Perplexity = exp(loss)
Perplexity < 10:  Excellent text modeling
Perplexity 10-30: Good performance
Perplexity 30-100: Acceptable performance
Perplexity > 100: Poor modeling

Generated Text Quality

Low Temperature (0.5): More conservative, coherent text
Medium Temperature (1.0): Balanced creativity and coherence
High Temperature (1.5): More creative but potentially less coherent

Ablation Study Insights

Hidden Size: Larger hidden states capture more context
RNN Type: LSTM typically outperforms vanilla RNN and GRU
Layers: 2-3 layers usually optimal for this task size

Troubleshooting

Common Issues

1. Memory Errors

# Reduce batch size
train_loader = DataLoader(dataset, batch_size=16, shuffle=True, num_workers=0)

# Reduce sequence length
dataset = TextDataset(texts, vocab, sequence_length=25)

2. Slow Convergence

# Increase learning rate
optimizer = optim.Adam(model.parameters(), lr=0.003)

# Use gradient clipping
nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

3. Poor Text Generation

# Train for more epochs
train_losses = train_model(model, train_loader, criterion, optimizer, num_epochs=20)

# Increase model capacity
model = RNNModel(vocab_size, hidden_size=512, num_layers=3)

4. Gradient Explosion

# Already implemented in training loop
nn.utils.clip_grad_norm_(model.parameters(), clip_grad=5.0)

Performance Optimization

For CPU Training

# Reduce batch size and sequence length
batch_size = 16
sequence_length = 30

# Use simpler model
model = RNNModel(vocab_size, hidden_size=128, num_layers=2)

For GPU Training

# Check GPU availability
if torch.cuda.is_available():
    device = torch.device('cuda')
    print(f"Using GPU: {torch.cuda.get_device_name()}")
    
    # Increase batch size for GPU
    batch_size = 64
    sequence_length = 50

Code Structure

Main Components

TextDataset Class: Handles text preprocessing and sequence generation
RNNModel Class: The neural network architecture
load_shakespeare_data(): Loads built-in Shakespeare text
build_vocabulary(): Creates word-to-index mappings
train_model(): Training loop with gradient clipping
generate_text(): Text generation with temperature sampling
ablation_study(): Systematic parameter testing

Customization Options

# Custom text data
my_texts = ["Your custom text here...", "More text..."]
vocab = build_vocabulary(my_texts)
dataset = TextDataset(my_texts, vocab)

# Custom model architecture
model = RNNModel(
    vocab_size=len(vocab),
    embedding_dim=256,      # Larger embeddings
    hidden_size=512,        # More hidden units
    num_layers=3,           # Deeper network
    dropout=0.3,            # Less dropout
    rnn_type='GRU'          # Different RNN type
)

# Custom generation parameters
generated = generate_text(
    model, dataset,
    seed_text="Custom seed",
    max_length=50,          # Longer generation
    temperature=0.7         # Control randomness
)

Assignment Deliverables

Required Files

Jupyter Notebook (.ipynb) or Python script (.py): rnn_shakespeare_clean.py
PDF Report (LaTeX): Use results from this implementation
GPT Prompts (.txt): Document all prompts used

Report Sections

Dataset and Preprocessing: Shakespeare text handling and vocabulary
Model Architecture: RNN design choices and implementation
Training Results: Loss curves, perplexity progression
Text Generation: Examples of generated text with different seeds
Ablation Study: Comparison of different hyperparameters
Conclusion: Best configuration and performance analysis

Expected Performance

Baseline Results

Validation Perplexity: 15-30
Training Time: 5-15 minutes (CPU), 2-5 minutes (GPU)
Generated Text Quality: Coherent short phrases

Performance Benchmarks

Model Configuration	Perplexity	Training Time	Quality
RNN (128 hidden)	40-60	5 min	Basic
LSTM (256 hidden)	15-25	10 min	Good
LSTM (512 hidden)	12-20	15 min	Excellent
GRU (256 hidden)	18-28	8 min	Good

Text Generation Examples

# Good generation (perplexity < 20):
"To be or not to be that is the question of life and death and love"

# Excellent generation (perplexity < 15):
"Romeo Romeo wherefore art thou Romeo deny thy father and refuse thy name for love"

References

LSTM: Hochreiter & Schmidhuber (1997)
GRU: Cho et al. (2014)
Text Generation: Karpathy (2015)
Shakespeare Dataset: Project Gutenberg

Tips for Success

Start with LSTM: Usually performs better than vanilla RNN
Monitor Perplexity: Lower values indicate better text modeling
Use Gradient Clipping: Prevents exploding gradients in RNNs
Temperature Tuning: 0.7-1.0 often gives best generation quality
Sufficient Training: 10+ epochs usually needed for coherent text
Vocabulary Size: Balance between coverage and complexity

Understanding the Output

Training Curves

Decreasing Loss: Model is learning text patterns
Plateau: May need more epochs or different hyperparameters
Oscillation: Learning rate might be too high

Generated Text Analysis

Repetition: May indicate overfitting or insufficient diversity
Incoherence: Model might need more training or capacity
Shakespeare Style: Good performance shows learned patterns

Perplexity Interpretation

Perplexity: Measures how "surprised" the model is by the text
Lower is Better: Less surprise means better text modeling
Rule of Thumb: Perplexity < 30 for good text generation

For questions or issues, refer to the troubleshooting section or check the PyTorch RNN documentation.

Name		Name	Last commit message	Last commit date
Code		Code
LICENSE		LICENSE
README.md		README.md

Latest commit

History

Folders and files

Repository files navigation