Sequence to Sequence Models

Recurrent Neural Networks (RNNs) for Language Modeling with Keras

David Cecchini

Instructor

Sequence to sequence

Possible architectures:

Many inputs with one output
- Sentiment analysis
- Classification
Many inputs to many outputs
- Text generation
- Neural Machine Translation (NMT)

# Pre-trained model
model.generate_sheldon_phrase()

'knock knock. penny. do you have an epost is part in your expert, 
too bealie to play the tariment with last night.'

How to build text generation models:

Decide if a token will be characters or words
- Words demands very large datasets (hundred of millions sentences)
- Chars can be trained faster, but can generate typos
Prepare the data
- Build training sample with (past tokens, next token) examples
Design the model architecture
- Embedding layer, number of layers, etc.
Train and experiment

Neural Machine Translation: example

# Pre-trained model
model.translate("Vamos jogar futebol?")

'Let's go play soccer?'

How to build NMT models:

Get a sample of translated sentences
- For example, the Anki project
Prepare the data
- Tokenize input language sentences
- Tokenize output language sentences
Design the model architecture
- Encoder and decoder
Train and experiment

In this chapter:

Text Generation
- Use pre-trained model to generate a sentence
- Learn to prepare the data and build the model
Neural Machine Translation (NMT)
- All-in-one NMT model

Recurrent Neural Networks (RNNs) for Language Modeling with Keras