Pre-trained models for text generation

Deep Learning for Text with PyTorch

Shubham Jain

Instructor

Why pre-trained models?

Benefits

  1. Trained on extensive datasets
  2. High performance across various text generation tasks
    • Sentiments analysis
    • Text completion
    • Language translation

Limitations

  1. High computational cost for training
  2. Large storage requirements
  3. Limited customization options
Deep Learning for Text with PyTorch

Pre-trained models in PyTorch

  • Hugging Face Transformers: library of pre-trained models
  • Pre-trained models:
    • GPT-2
    • T5

HuggingFace Logo

Deep Learning for Text with PyTorch

Understanding GPT-2 Tokenizer and Model

GPT2LMHeadModel:

  • HuggingFace's take on GPT-2
  • Tailored for text generation

GPT2Tokenizer:

  • Converts text into tokens
  • Handles subword tokenization: 'larger' might become ['large', 'r']
Deep Learning for Text with PyTorch

GPT-2: text generation implementation

import torch
from transformers import GPT2Tokenizer, GPT2LMHeadModel

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2')

seed_text = "Once upon a time"

input_ids = tokenizer.encode(seed_text, return_tensors='pt')
Deep Learning for Text with PyTorch

GPT-2: text generation implementation II

output = model.generate(

                                                           )
Deep Learning for Text with PyTorch

GPT-2: text generation implementation II

output = model.generate(input_ids, max_length=40, 

                                                           )
Deep Learning for Text with PyTorch

GPT-2: text generation implementation II

output = model.generate(input_ids, max_length=40, temperature=0.7,     

                                                           )
Deep Learning for Text with PyTorch

GPT-2: text generation implementation II

output = model.generate(input_ids, max_length=40, temperature=0.7,     
                        no_repeat_ngram_size=2, 
                                                           )
Deep Learning for Text with PyTorch

GPT-2: text generation implementation II

output = model.generate(input_ids, max_length=40, temperature=0.7,     
                        no_repeat_ngram_size=2, 
                        pad_token_id=tokenizer.eos_token_id)
Deep Learning for Text with PyTorch

GPT-2: text generation output

generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
print(generated_text)

Generated Text: Once upon a time, the world was a place of great beauty
and great danger. The world of the gods was the place where the great gods were 
born, and where they were to live.
Deep Learning for Text with PyTorch

T5: Language translation implementation

  • t5-small: Text-to-Text Transfer Transformer
  • Pretrained model for language translation tasks
import torch
from transformers import T5Tokenizer, T5ForConditionalGeneration

tokenizer = T5Tokenizer.from_pretrained("t5-small")
model = T5ForConditionalGeneration.from_pretrained("t5-small")

input_prompt = "translate English to French: 'Hello, how are you?'"

input_ids = tokenizer.encode(input_prompt, return_tensors="pt")

output = model.generate(input_ids, max_length=100)
Deep Learning for Text with PyTorch

T5: Language translation output

generated_text = tokenizer.decode(output[0], skip_special_tokens=True)

print("Generated text:",generated_text)

Generated text:
"Bonjour, comment êtes-vous?"
Deep Learning for Text with PyTorch

Choosing the right pre-trained model

  • Many exist!

 

  • GPT-2: Text generation
  • DistilGPT-2 (Smaller version of GPT-2): Text generation
  • BERT: Text classification, question-answering
  • T5 (t5-small is the smaller version of T5): Language translation, summarization

 

  • Find them in HuggingFace and other repositories
Deep Learning for Text with PyTorch

Let's practice!

Deep Learning for Text with PyTorch

Preparing Video For Download...