Natural Language Processing (NLP) basics

Natural Language Processing with spaCy

Azadeh Mobasher

Principal Data Scientist

Natural Language Processing (NLP)

A subfield of Artificial Intelligence (AI)
Helps computers to understand human language
Helps extract insights from unstructured data
Incorporates statistics, machine learning models and deep learning models

NLP, a subfield of AI

Sentiment analysis

Use of computers to determine the underlying subjective tone of a piece of writing

Sentiment Analysis Examples

Named entity recognition (NER)

Locating and classifying named entities mentioned in unstructured text into pre-defined categories
Named entities are real-world objects such as a person or location

NER Examples

Chatbots

spaCy is a free, open-source library for NLP in Python which:

spaCy and NLP

As the first step, spaCy can be installed using the Python package manager pip
spaCy trained models can be downloaded
Multiple trained models are available for English language at spacy.io

$ python3 pip install spacy

python3 -m spacy download en_core_web_sm
import spacy
nlp = spacy.load("en_core_web_sm")

Loaded spaCy model en_core_web_sm = nlp object
nlp object converts text into a Doc object (container) to store processed text

Text processing with spaCy

import spacy
nlp = spacy.load("en_core_web_sm")
text = "A spaCy pipeline object is created."
doc = nlp(text)

Tokenization
- A Token is defined as the smallest meaningful part of the text.
- Tokenization: The process of dividing a text into a list of meaningful tokens

print([token.text for token in doc])

['A', 'spaCy', 'pipeline', 'object', 'is', 'created', '.']

Natural Language Processing with spaCy