Named entity recognition on transcribed text

Spoken Language Processing in Python

Daniel Bourke

Machine Learning Engineer/YouTube Creator

Installing spaCy

# Install spaCy
$ pip install spacy

# Download spaCy language model
$ python -m spacy download en_core_web_sm

Using spaCy

import spacy


# Load spaCy language model
nlp = spacy.load("en_core_web_sm")

# Create a spaCy doc
text = ("I'd like to talk about a smartphone I ordered on July 31st "
        "from your Sydney store, my order number is 4093829. I spoke "
        "to one of your customer service team, Georgia, yesterday.")
doc = nlp(text)

spaCy tokens

# Show different tokens and positions
for token in doc:
  print(token.text, token.idx)

I 0
'd 1
like 4
to 9
talk 12
about 17
a 23
smartphone 25...

spaCy sentences

# Show sentences in doc
for sentence in doc.sents:
  print(sentence)

I'd like to talk about a smartphone I ordered on July 31st from your Sydney store, 
my order number is 4093829.
I spoke to one of your customer service team, Georgia, yesterday.

spaCy named entities

Some of spaCy's built-in named entities:

PERSON People, including fictional.
ORG Companies, agencies, institutions, etc.
GPE Countries, cities, states.
PRODUCT Objects, vehicles, foods, etc. (Not services.)
DATE Absolute or relative dates or periods.
TIME Times smaller than a day.
MONEY Monetary values, including unit.
CARDINAL Numerals that do not fall under another type.

spaCy named entities

# Find named entities in doc
for entity in doc.ents:
  print(entity.text, entity.label_)

July 31st DATE
Sydney GPE
4093829 CARDINAL
one CARDINAL
Georgia GPE
yesterday DATE

Custom named entities

# Check spaCy pipeline
print(nlp.pipe_names)

['tok2vec', 'tagger', 'parser',
 'attribute_ruler', 'lemmatizer', 'ner']

Changing the pipeline

# Add entity_ruler to the pipeline, before ner
ruler = nlp.add_pipe("entity_ruler", before="ner")

# Add token pattern to ruler
ruler.add_patterns([{"label":"PRODUCT", "pattern": "smartphone"}])

# Check updated pipeline
nlp.pipe_names

Changing the pipeline

['tok2vec', 'tagger', 'parser', 'attribute_ruler',
 'lemmatizer', 'entity_ruler', 'ner']

Testing the new pipeline

# Recreate the doc with the updated pipeline
doc = nlp(text)

# Test new entity rule
for entity in doc.ents:
  print(entity.text, entity.label_)

smartphone PRODUCT
July 31st DATE
Sydney GPE
4093829 CARDINAL
one CARDINAL
Georgia GPE
yesterday DATE

Let's rocket and practice spaCy!

Spoken Language Processing in Python