Sentence autocompletion using Encoder-Decoder

Natural Language Generation in Python

Biswanath Halder

Data Scientist

Encoder decoder architecture

Encoder
- Summarizes input information is states.
- Outputs ignored.
Decoder
- Initial state - final state of the encoder.
- Final states ignored.
- Input during training - original target.
- Input during inference - predicted target.

Encoder Decoder Block Diagram

Encoder-Decoder Sentence Autocomplete Training

# Create the input layer of the encoder
encoder_input = Input(shape=(None, len(vocabulary)))

# Create LSTM Layer of size 256
encoder_LSTM = LSTM(256, return_state = True)

# Save encoder output, hidden and cell state
encoder_outputs, encoder_h, encoder_c = encoder_LSTM(encoder_input)

# Save encoder states
encoder_states = [encoder_h, encoder_c]

decoder_input = Input(shape=(None, len(vocabulary)))

decoder_LSTM = LSTM(256, return_sequences=True, return_state = True)

decoder_out, _ , _ = decoder_LSTM(decoder_input,
                                  initial_state=encoder_states)

decoder_dense = Dense(len(vocabulary), activation='softmax')

decoder_out = decoder_dense(decoder_out)

model = Model(inputs=[encoder_input, decoder_input],
                          outputs=[decoder_out])

model.summary()

model.compile(optimizer='adam', loss='categorical_crossentropy')

model.fit(x=[input_data_prefix, input_data_suffix], y=target_data,
                  batch_size=64, epochs=1, validation_split=0.2)

Natural Language Generation in Python