Interpretable and Explainable NER With LIME

While much progress has been made to develop newer and more recent deep learning models with gazillion parameters, little effort has been made to explain the output of these models.

During a workshop in December 2020, Gradio CEO Abubakar Abed examined the way GPT-3 generates a text about religions using the spot phrase, “Two _ enter a. When observing the first ten responses of various religions, it was found that GPT-3 mentioned violence once for Jews, Buddhists, and Sikhs, twice for Christians, but Nine out of ten for Muslims.”

Subsequently, Abid’s team showed that inserting a positive text about Muslims into a large language model reduced the number of signs of violence about Muslims by nearly 40%. Even the creator of GPT-3, OpenAI, released a paper in May 2020 with tests that found that GPT-3 generally has a low opinion of black people and exhibits sexism and other forms of bias. Examples of the kind of societal bias embedded in these large language paradigms abound, from racist statements to toxic content.

Deep learning models are like a black box; Give it an input and it gives you an output without explaining the reason for the decision whether it is text classification, text generation or Named Entity Recognition (NER). It is crucial to closely monitor the output of this model and, more importantly, to be able to explain the decision-making process of these models. Explaining the logic behind the output would give us more confidence to trust or not to trust the model’s prediction.

Explanation of NER models using LIME

In this tutorial, we will focus on explaining the prediction of a named entity recognition model using LIME (Local Neutral Interpretation Model Explanations). You can learn more from the original paper.

LIME is a neutral model, which means that it can be applied to explain any type of model output without reaching its climax. It does this by obfuscating local features around target prediction and output measurement. In our specific case, we will change the tokens around the target entity, and then try to scale the model’s output.

Below is an explanation of how LIME works.

Here’s an explanation from the LIME website “The archetype’s decision function is represented by a blue/pink background and is obviously non-linear. The bright red cross is the example shown (let’s call it X). We sample the perturbed states around X, and weight them according to their proximity to X. (The weight here is represented by the volume.) We get the archetype prediction about these perturbed states, and then learn a linear model (dashed line) that approximates well to the model near X. Note that the explanation, in this case, is not globally faithful, but true locally about X.”

LIME process source

LIME outputs a list of tokens with a contribution score in the model prediction (see example below for text classification). This provides local interpretation, and also allows determining which feature changes will have the greatest impact on the prediction.

Explanation of the document class created by LIME. source

In this tutorial, we will focus on explaining NER model output using LIME.

Here are the steps:

  • Generate training data for our model

  • Training the NER model on our custom annotated dataset

  • Select the target word to explain

Download data

In this tutorial, we will train the NER model that predicts skills, experience, diploma and specialist diploma from job descriptions. Data was obtained from Kaggle. Refer to this article for more details about the data annotation part using UBIAI.

To train the NER model, we will use the CRF algorithm because it can easily output confidence scores for each predicted entity which is essential for LIME work.

The first step is to upload the annotated data to our notebook; The data is formatted in IOB format.

This is a small sample:

      years I-EXPERIENCE
      experience O
      in O
      the O
      online B-SKILLS
      advertising I-SKILLS
      or O
      research B-SKILLS

Next, we import some packages and preprocess our data in the bucket list (token, tag):

! pip install -U 'scikit-learn<0.24'
      !pip install sklearn_crfsuite #Installing CRF
      !pip install eli5 # Installing Lime

import matplotlib.pyplot as plt'ggplot')
      from itertools import chain
      import nltk
      import sklearn
      import scipy.stats
      from sklearn.metrics import make_scorer
      #from sklearn.cross_validation import cross_val_score
      from sklearn.model_selection import RandomizedSearchCV
      import sklearn_crfsuite
      from sklearn_crfsuite import scorers
      from sklearn_crfsuite import metrics
      import random
      def import_documents_set_iob(train_file_path):
          with open(train_file_path,  encoding="utf8") as f:
              tokens_in_file = f.readlines()
          # construct list of list train set format
          new_train_set = []
          for index_token,token in enumerate(tokens_in_file):
              # detect new document
              is_new_document = False
              if token == '-DOCSTART- -X- O O
                  # So, there's a new document
                  is_new_document = True
                  document = []
                  # A document is a set (triplets) of token name, POS token, tag token
                  split_token = token.split("    ")
                  try :
                      #print ("except :",split_token)
                      # if end of document, we store the document in th train set
                      if (tokens_in_file[index_token+1] == '-DOCSTART- -X- O O
' ):
                      # detect the end of file or the end of all tokens in all documents in train set
                      if (index_token== (len(tokens_in_file) - 1)) :
          return new_train_set


Let’s see what the menu looks like:

train_file_path = r"/content/train_data.tsv"
      train_sents = import_documents_set_iob(train_file_path)
      #Small sample of the output
      ('of', 'O'), ('advanced', 'B-SKILLS'), ('compute', 'I-SKILLS'), ('and', 'O')

Data preprocessing

To train the CRF model, we need to convert our annotated text into numerical features. For more information, check out the CRF documentation:

# Utils functions to extract features
      def word2features(sent, i):
          word = sent[i][0]
          #postag = sent[i][1]
          features = {
              'bias': 1.0,
              'word.lower()': word.lower(),
              'word[-3:]': word[-3:],
              'word[-2:]': word[-2:],
              'word.isupper()': word.isupper(),
              'word.istitle()': word.istitle(),
              'word.isdigit()': word.isdigit(),
              #'postag': postag,
              #'postag[:2]': postag[:2],
          if i > 0:
              word1 = sent[i-1][0]
              postag1 = sent[i-1][1]
                  '-1:word.lower()': word1.lower(),
                  '-1:word.istitle()': word1.istitle(),
                  '-1:word.isupper()': word1.isupper(),
                  #'-1:postag': postag1,
                  #'-1:postag[:2]': postag1[:2],
              features['BOS'] = True
          if i < len(sent)-1:
              word1 = sent[i+1][0]
              postag1 = sent[i+1][1]
                  '+1:word.lower()': word1.lower(),
                  '+1:word.istitle()': word1.istitle(),
                  '+1:word.isupper()': word1.isupper(),
                  #'+1:postag': postag1,
                  #'+1:postag[:2]': postag1[:2],
              features['EOS'] = True
          return features
      def sent2features(sent):
          return [word2features(sent, i) for i in range(len(sent))]
      def sent2labels(sent):
          #return [label for token, postag, label in sent]
          return [label for token,  label in sent]
      def sent2tokens(sent):
          #return [token for token, postag, label in sent]
          return [token for token, label in sent]
      print ("example extracted features from single word :",sent2features(train_sents[0])[0])

After that’s done, we’re ready to train – we just need to put the training/testing features and target labels into their respective menus:

X_train = [sent2features(s) for s in train_sents]
      y_train = [sent2labels(s) for s in train_sents]
      X_test = [sent2features(s) for s in test_sents]
      y_test = [sent2labels(s) for s in test_sents]

model training

We start the training with 100 repetitions:

crf = sklearn_crfsuite.CRF(
   , y_train)


After training, we obtained an F-1 score of 0.61 which is not high but is reasonable given the amount of annotated data set. Scores for each entity:

sorted_labels = sorted(
      key=lambda name: (name[1:], name[0])
      y_test, y_pred, labels=sorted_labels, digits=3


Image source: rating agencies after training

NER interpretable with LIME

Now that the model has been trained, we are ready to explain its naming predictions using the LIME algorithm. First, we initialize our NERExplainerGenerator class which will generate features from the input text and fetch them into our form:

from eli5.lime import TextExplainer
      from eli5.lime.samplers import MaskingTextSampler
      from nltk.tokenize import word_tokenize'punkt')
      import numpy as np
      class NERExplainerGenerator(object):
          def __init__(self, model):
              self.model = model
          def sents2tuples(self,sents):
            res = []
            for sent in sents:
              tokens = word_tokenize(sent)
              res.append([(token,'') for token in tokens])
            return res
          def _preprocess(self, texts):
            texts = [res for res in self.sents2tuples(texts)]
            X = [sent2features(s) for s in texts]
            return X
          def dict2vec(self,pred):
              vectors = []
              for sent in pred:
                sent_res = []
                for dic in sent:
                  vector = [dic[key] for key in self.model.classes_]
                sent_res = np.array(sent_res)
              vectors = np.array(vectors)
              return vectors
          def get_predict_function(self, word_index):
              def predict_func(texts):
                  X = self._preprocess(texts)
                  pred = self.model.predict_marginals(X)
                  pred = self.dict2vec(pred)
                  return pred[:,word_index,:]
              return predict_func

We will test using the following sentence from the job description:

text=""'6+ years of Web UI/UX design experience
      Proven mobile web application design experience'''
      explainer= NERExplainerGenerator(crf)
      for index,word in enumerate(word_tokenize(text)):
      0 6+
      1 years
      2 of
      3 Web
      4 UI/UX
      5 design
      6 experience
      7 Proven
      8 mobile
      9 web
      10 application
      11 design
      12 experience

Finally, we need to set up the LIME annotation algorithm. Here’s what each function means:

  • MaskingTextSampler: If you remember back in the introduction we mentioned that LIME will try to obfuscate local features and record our model’s output. It does this by randomly replacing 70% of the tokens with a “UNK” token. The percentage can be adjusted if needed, but 70 is the default value.

  • Samples and Similarities: The LIME model will generate many sentences by randomizing the original sentence with the “UNK” symbol. Here are some examples.

    [‘6+ years of UNK UNK/UX design experience Proven UNK web UNK UNK experience’, ‘UNK+ years UNK Web UI/UX design experience Proven mobile web application UNK UNK’, ‘6+ UNK of Web UI/UX design experience Proven UNK web application UNK experience’, ‘UNK+ years of Web UI/UNK UNK UNK UNK mobile web application UNK experience’]

For each sentence, we will have a label expected from our NER model. LIME will then train on the data with a white linear model that explains the contribution of each token: (text, func)

Let’s for example try to explain the naming of the word “UI/UX” which contains a file word index = 4:

word_index = 4 #explain UI/UX label
      func = explainer.get_predict_function(word_index)
      sampler = MaskingTextSampler(
      samples, similarity = sampler.sample_near(text, n_samples=4)
      te = TextExplainer(
   , func)
      #the explainer needs just the one instance text from texts list
      explain = te.explain_prediction(
      print("WORD TO EXPLAIN", word_tokenize(text)[word_index])


Here is the output:

Image source: tartar output

Green colors mean the token has a positive contribution to the expected label and red means the token has a negative contribution. The model correctly predicted, at 0.95 probability, that “UI/UX” is part of the I-SKILLS multi-symbol skill. The word “web” was a strong indication of the expected nomenclature. In agreement with the first statement, the B-SKILLS designation has a lower probability of 0.018 with a strong negative contribution of the word “web”.

Lime also provides a contribution for each token:

Author image: a contribution for each feature

We note that the web, design and years have the highest contribution to the expected mark I-SKILLS.


As we move towards large and complex back-end AI models, understanding the decision-making behind the predictions is of paramount importance so that we can trust the model’s output.

In this tutorial, we show how to train a custom NER model and explain its output using the LIME algorithm. LIME is a neutral model and can be applied to explain the output of any complex models whether it is image recognition, text classification or NER as in this tutorial.

If you have any questions or want to create custom forms for your specific case, leave a note below or send us an email at

Follow us on Twitter UBIAI5


Leave a Comment