Identify Person, Place and Organisation in content using Python

Sidharth Macherla 8 Comments , ,
This article outlines the concept and python implementation of Named Entity Recognition using StanfordNERTagger. The technical challenges such as installation issues, version conflict issues, operating system issues that are very common to this analysis are out of scope for this article.

NER NLP using Python

Table of contents:

1. Named Entity Recognition defined
2. Business Use cases
3. Installation Pre-requisites
4. Python Code for implementation
5. Additional Reading: CRF model, Multiple models available in the package
6. Disclaimer

1. Named Entity Recognition Defined
The process of detecting and classifying proper names mentioned in a text can be defined as Named Entity Recognition (NER). In simple words, it locates person name, organization and location etc. in the content. This is generally the first step in most of the Information Extraction (IE) tasks of Natural Language Processing.
NER Sample

2. Business Use Cases

There is a need for NER across multiple domains. Below are a few sample business use cases for your reference.
  1. Investment research: To identify the various announcements of the companies, people’s reaction towards them and its impact on the stock prices, one needs to identify people and organisation names in the text
  2. Chat-bots in multiple domains: To identify places and dates for booking hotel rooms, air tickets etc.
  3. Insurance domain: Identify and mask people’s names in the feedback forms before analyzing. This is needed for being regulatory compliant(example: HIPAA)

3. Installation Prerequisites
2. Unzip the zipped folder and save in a drive.
3. Copy the “stanford-ner.jar” from the folder and save it just outside the folder as shown in the image
4. Download the caseless models from https://stanfordnlp.github.io/CoreNLP/history.html by clicking on “caseless” as given below. The models in the first link work as well. However, the caseless models help in identifying named entities even when they are not capitalised as required by formal grammar rules. 
5. Save the folder in the same location as the Stanford NER folder for ease of access
Stanford NER Installation - Step1





NER Installation - Step2







4. Python Code for implementation:
#Import all the required libraries.
import os
from nltk.tag import StanfordNERTagger
import pandas as pd

#Set environmental variables programmatically.
#Set the classpath to the path where the jar file is located
os.environ['CLASSPATH'] = "<path to the file>/stanford-ner-2015-04-20/stanford-ner.jar"

#Set the Stanford models to the path where the models are stored
os.environ['STANFORD_MODELS'] = '<path to the file>/stanford-corenlp-caseless-2015-04-20-models/edu/stanford/nlp/models/ner'

#Set the java jdk path
java_path = "C:/Program Files/Java/jdk1.8.0_161/bin/java.exe"
os.environ['JAVAHOME'] = java_path


#Set the path to the model that you would like to use
stanford_classifier  =  '<path to the file>/stanford-corenlp-caseless-2015-04-20-models/edu/stanford/nlp/models/ner/english.all.3class.caseless.distsim.crf.ser.gz'

#Build NER tagger object
st = StanfordNERTagger(stanford_classifier)

#A sample text for NER tagging
text = 'srinivas ramanujan went to the united kingdom. There he studied at cambridge university.'

#Tag the sentence and print output
tagged = st.tag(str(text).split())
print(tagged)

Output
[(u'srinivas', u'PERSON'), 
(u'ramanujan', u'PERSON'), 
(u'went', u'O'), 
(u'to', u'O'), 
(u'the', u'O'), 
(u'united', u'LOCATION'), 
(u'kingdom.', u'LOCATION'), 
(u'There', u'O'), 
(u'he', u'O'), 
(u'studied', u'O'), 
(u'at', u'O'), 
(u'cambridge', u'ORGANIZATION'), 
(u'university', u'ORGANIZATION')]

5. Additional Reading

StanfordNER algorithm leverages a general implementation of linear chain Conditional Random Fields (CRFs) sequence models. CRFs seem very similar to Hidden Markov Model but are very different.

Below are some key points to note about the CRFs in general.
  1. It is a discriminative model unlike the HMM model and thus models the conditional probability
  2. It does not assume independence of features unlike the HMM model. This means that the current word, previous word, next word are all considered for model as features
  3. Relative to HMM or Max ent Markov Models, CRFs are the slowest

6. Disclaimer
This article explains the implementation of StanfordNER algorithm for research purposes and does not promote it for commercial gains. For any questions on commercial aspects of implementing this algorithm, please contact Stanford University
Related Posts
Spread the Word!
Share
About Author:
Sidharth Macherla

Sidharth Macherla has over 12 years of experience in data science and his current area of focus is Natural Language Processing . He has worked across Banking, Insurance, Investment Research and Retail domains.

Let's Get Connected: LinkedIn

8 Responses to "Identify Person, Place and Organisation in content using Python"
  1. Good article. But I was not able to download a folder but only an executable from https://stanfordnlp.github.io/CoreNLP/history.html

    But where can I find the folder like you have shown in the image ?

    ReplyDelete
    Replies
    1. Hi Deepsss,
      The jar file is an archive folder of sort. If you right click on the jar file and extract the folders inside, you will be able to find the caseless models. Here is the path to the folder. "\stanford-corenlp-caseless-2015-04-20-models.jar\edu\stanford\nlp\models\ner\"

      Delete
  2. getting error while doing import pandas

    ReplyDelete
  3. I have given the right path still i am getting the look up error:NLTK was unable to find the D:\CVParser\stanford-corenlp-caseless-2015-04-20-models\edu\stanford
    lp\models
    er\english.all.3class.caseless.distsim.crf.ser.gz file!
    Use software specific configuration paramaters or set the STANFORD_MODELS environment variable.

    ReplyDelete
  4. Not working with latest packages for the name SUFIYAN AHMED works in Technology. Above 2015 models are working fine but not working with latest models

    ReplyDelete
  5. Great post!!Just a quick doubt, It is identifying Houston as location but not Santa Clara. Santa Clara is splitting up and showing as person. Any suggestions?

    ReplyDelete
  6. Very useful article. Thanks a lot for sharing.

    ReplyDelete
Next → ← Prev