Data Structures: Doc, Span and Token

Advanced NLP with spaCy

Ines Montani

spaCy core developer

The Doc object

# Create an nlp object
from spacy.lang.en import English
nlp = English()

# Import the Doc class
from spacy.tokens import Doc

# The words and spaces to create the doc from
words = ['Hello', 'world', '!']
spaces = [True, False, False]

# Create a doc manually
doc = Doc(nlp.vocab, words=words, spaces=spaces)

The Span object (1)

Illustration of a Span object within a Doc with token indices

The Span object (2)

# Import the Doc and Span classes
from spacy.tokens import Doc, Span

# The words and spaces to create the doc from
words = ['Hello', 'world', '!']
spaces = [True, False, False]

# Create a doc manually
doc = Doc(nlp.vocab, words=words, spaces=spaces)

# Create a span manually
span = Span(doc, 0, 2)

# Create a span with a label
span_with_label = Span(doc, 0, 2, label="GREETING")

# Add span to the doc.ents
doc.ents = [span_with_label]

Best practices

Doc and Span are very powerful and hold references and relationships of words and sentences
- Convert result to strings as late as possible
- Use token attributes if available – for example, token.i for the token index
Don't forget to pass in the shared vocab

Let's practice!

Advanced NLP with spaCy