Generating translations from the model

Machine Translation with Keras

Thushan Ganegedara

Data Scientist and Author

Previous model vs new model

Encoder decoder repeat vector

Encoder decoder which input

Trained model

Teacher Forcing trained model

Decoder of the inference model

Takes in
- A onehot encoded word
- A state input (gets the state from previous timestep)
Produces
- A new state
- A prediction (i.e. a word)
Recursively feed the predicted word and the state back to the model as inputs

Full inference model

Inference model with the recursive decoder

Encoder recursive decoder

Inference model from the previous chapter

Value of sos and eos tokens

sos marks beginning of a translation (i.e. a French sentence).
- Feed in sos as the first word to the decoder and keep predicting
eos marks the end of a translation.
- Predictions stop when the word predicted by the model is eos
As a safety measure use a maximum length the model can predict for

Defining the generator encoder

Importing layers and Model

# Import Keras layers
import tensorflow.keras.layers as layers
from tensorflow.keras.models import Model

Defining model layers

en_inputs = layers.Input(shape=(en_len,en_vocab))
en_gru = layers.GRU(hsize, return_state=True)
en_out, en_state = en_gru(en_inputs)

Defining Model object

encoder = Model(inputs=en_inputs, outputs=en_state)

Defining the generator decoder

Defining the decoder Input layers

de_inputs = layers.Input(shape=(1, fr_vocab))
de_state_in = layers.Input(shape=(hsize,))

Defining the decoder's interim layers

de_gru = layers.GRU(hsize, return_state=True)
de_out, de_state_out = de_gru(de_inputs, initial_state=de_state_in)

de_dense = layers.Dense(fr_vocab, activation='softmax')
de_pred = de_dense(de_out)

Defining the decoder Model

decoder = Model(inputs=[de_inputs, de_state_in], outputs=[de_pred, de_state_out])

Copying the weights

Get weights of the layer l1
- w = l1.get_weights()
Set the weights of the layer l2 with w
- l2.set_weights(w)
In our model, there are three layers with weights
- Encoder GRU, Decoder GRU and Decoder Dense

en_gru_w = tr_en_gru.get_weights()
en_gru.set_weights(en_gru_w)

Which can also be written as,

en_gru.set_weights(tr_en_gru.get_weights())

Generating translations

en_sent = ['the united states is sometimes chilly during 
           december , but it is sometimes freezing in june .']

Converting the English sentence to a sequence

en_seq = sents2seqs('source', en_st, onehot=True, reverse=True)

Getting the context vector

de_s_t = encoder.predict(en_seq)

Converting "sos" (initial word to the decoder) to a sequence

de_seq = word2onehot(fr_tok, 'sos', fr_vocab)

Generating translations

fr_sent = ''

for _ in range(fr_len):      
  de_prob, de_s_t = decoder.predict([de_seq,de_s_t])

  de_w = probs2word(de_prob, fr_tok)

  de_seq = word2onehot(fr_tok, de_w, fr_vocab)

  if de_w == 'eos': break
  fr_sent += de_w + ' '

Time to translate!

Machine Translation with Keras