The training loop

Advanced NLP with spaCy

Ines Montani

spaCy core developer

The steps of a training loop

Loop for a number of times.
Shuffle the training data.
Divide the data into batches.
Update the model for each batch.
Save the updated model.

Recap: How training works

Diagram of the training process

Training data: Examples and their annotations.
Text: The input text the model should predict a label for.
Label: The label the model should predict.
Gradient: How to change the weights.

Example loop

TRAINING_DATA = [
    ("How to preorder the iPhone X", {'entities': [(20, 28, 'GADGET')]})
    # And many more examples...
]

# Loop for 10 iterations
for i in range(10):

    # Shuffle the training data
    random.shuffle(TRAINING_DATA)

    # Create batches and iterate over them
    for batch in spacy.util.minibatch(TRAINING_DATA):

        # Split the batch in texts and annotations
        texts = [text for text, annotation in batch]
        annotations = [annotation for text, annotation in batch]

        # Update the model
        nlp.update(texts, annotations)

# Save the model
nlp.to_disk(path_to_model)

Updating an existing model

Improve the predictions on new data
Especially useful to improve existing categories, like PERSON
Also possible to add new categories
Be careful and make sure the model doesn't "forget" the old ones

Setting up a new pipeline from scratch

# Start with blank English model
nlp = spacy.blank('en')

# Create blank entity recognizer and add it to the pipeline
ner = nlp.create_pipe('ner')
nlp.add_pipe(ner)

# Add a new label
ner.add_label('GADGET')

# Start the training
nlp.begin_training()

# Train for 10 iterations
for itn in range(10):
    random.shuffle(examples)

    # Divide examples into batches
    for batch in spacy.util.minibatch(examples, size=2):
        texts = [text for text, annotation in batch]
        annotations = [annotation for text, annotation in batch]

        # Update the model
        nlp.update(texts, annotations)

Let's practice!

Advanced NLP with spaCy