The Text Generating Function

Recurrent Neural Networks (RNNs) for Language Modeling with Keras

David Cecchini

Data Scientist

Generating sentences

Sentence is determined by punctuation. For example, . (period), ! (exclamation) or ? (question).
- The punctuation marks need to be in the vocabulary.
There is a sentence token, e.g. <SENT> and </SENT>, that determines when a sentence begins and ends.
- Need to pre-process the data to insert the labels.

Generating sentences

sentence = ''

# Loop until end of sentence
while next_char != '.':

    # Predict next char: Get pred array in position 0
    pred = model.predict(X)[0]

    char_index = np.argmax(pred)

    next_char = index_to_char(char_index)

    # Concatenate to sentence
    sentence  = sentence + next_char

Probability scaling

Scale the probability distribution.

Temperature: name from physics
- Small values: makes prediction more confident
- Value equal to one: no scaling
- higher values: makes prediction more creative
- Hyper-parameter: Try different values to fit the predictions to your need

Probability scaling

def scale_softmax(softmax_pred, temperature=1.0):
    # Take the logarithm
    scaled_pred = np.log(softmax_pred) / temperature

    # Re-apply the exponential
    scaled_pred = np.exp(scaled_pred)

    # Build probability distribution
    scaled_pred = scaled_pred / np.sum(scaled_pred)

    # Simulate multinomial
    scaled_pred = np.random.multinomial(1, scaled_pred, 1)

    # Return simulated class
    return np.argmax(scaled_pred)

Let's practice!

Recurrent Neural Networks (RNNs) for Language Modeling with Keras