Named Entity Recognition

Tagging names, concepts or key phrases is a crucial task for natural language understanding pipelines. Prodigy lets you label NER training data or improve an existing model’s accuracy.

Quickstart

Choosing the right recipe and workflow

So you have an NER problem you want to solve, and data to annotate. And you want to get it done as efficiently as possible. But how do you pick the right workflow for your use case?

1. Fully manual: This is the most classic way of annotating data. You’re shown a raw text and highlight all entity spans in the text. At the end of the process, you export “gold-standard” data that you can train your model with. Fully manual annotation is often the best choice if you want to create a new dataset, but are starting completely from scratch without any existing resources you can leverage. In Prodigy, you can use this workflow with the ner.manual recipe.

2. Manual with suggestions from patterns: Fully manual annotation can easily get tedious, which leads to mistakes and inconsistent data. You’re often doing the same thing over and over – for instance, in your data, mentions of “New York” may pretty much always refer to the location. Instead of labelling it by hand every single time, you can use keyword lists and patterns describing the tokens you’re looking for and make Prodigy pre-highlight the candidates for you. Even if your patterns only help 50% of the time, that’s still 50% less work for you. In Prodigy, you can use this workflow with ner.manual and the --patterns option. Also see the docs on patterns.

3. Manual with suggestions from model: Instead of patterns, you can also use an existing model to highlight suggestions and save you time. This workflow is also useful if you want to train a model with a mix of new and existing categories. You can use the existing model to help you label the entity types you’re interested in, correct any mistakes it makes and add a new category on top. In Prodigy, you can use this workflow with the ner.correct recipe for spaCy pipeline or a custom recipe for any other model that predicts named entities.

4. Binary with active learning and a model in the loop: This workflow is useful if you already have a model and want to fine-tune it on more data. Instead of annotating every example, you can use the model to suggest you the most relevant examples to annotate and give it feedback on its predictions. There are many different ways you can select the “best” examples, and a whole line of research dedicated to exploring active learning techniques. Prodigy’s ner.teach recipe implements simple uncertainty sampling with beam search: for each example, the annotation model gets a number of analyses and asks you to accept or reject the entity analyses it’s most uncertain about. Based on your decisions, the model is updated in the loop and guided towards better predictions. Prodigy also includes utilities that let you implement custom workflows with a model in the loop.

Video tutorial: training a model to predict ingredients

The following video shows an end-to-end workflow for training a named entity recognition model to recognize food ingredients from scratch, taking advantage of semi-automatic annotation with ner.manual and ner.correct, as well as modern transfer learning techniques. It took around 2.5 hours to create 949 annotations, including 20% evaluation examples, which was enough to achieve 85% accuracy. You can download the raw and annotated datasets from GitHub.

Fully manual annotation

To get started with manual NER annotation, all you need is a file with raw input text you want to annotate and a spaCy pipeline for tokenization (so the web app knows what a word is and can allow more efficient highlighting). The following command will start the web server with the ner.manual recipe, stream in headlines from news_headlines.jsonl and provides the label options PERSON, ORG and PRODUCT.

Character/Token-based highlighting toggle New: 1.14.5

The ner.manual recipe also lets you set a --highlight-chars flag to allow highlighting individual characters instead of only tokens. As of v1.14.5 the flag adds a checkbox button to the UI for switching between character and character based highlighting while annotating.

Character-based highlighting will only store the character offsets of your annotation and won’t add a "tokens" property to the saved task.

Manual annotation with patterns

To annotate with patterns, you’ll need a patterns file, a spaCy pipeline for tokenization (or your own custom tokenizer) and a source of input text. In the following example, we used 100 match patterns for fashion brands. You can find the full code and data of this project in our projects repo.

fashion_brands_patterns.jsonl (excerpt)
{"label": "FASHION_BRAND", "pattern": [{"lower": "ann"}, {"lower": "taylor"}]}
{"label": "FASHION_BRAND", "pattern": [{"lower": "topshop"}]}
{"label": "FASHION_BRAND", "pattern": [{"lower": "naked"}, {"lower": "and"}, {"lower": "famous"}]}

Using the ner.manual, we can now stream in and annotate the data. All pattern matches will be pre-highlighted in the UI, so we only need to make corrections (unselect wrong or incomplete matches) and add the entities that the patterns missed.

In about 2 hours, we collected 1735 annotations (1235 training examples, 500 evaluation examples). The trained spaCy pipeline with word vectors and pretraining achieved 82.1% accuracy on the task.

Working with patterns

Match patterns are typically provided as a JSONL (newline-delimited JSON) file and can be used to pre-highlight spans for faster and more efficient annotation. If you have words and phrases that you know are pretty much always an entity, you can use patterns so you only have to annotate the exceptions. Prodigy supports two types of patterns:

patterns.jsonl
{"pattern": [{"lower": "new"}, {"lower": "york"}], "label": "GPE"}
{"pattern": "Berlin", "label": "GPE"}

1. Token patterns: These patterns are lists of dictionaries with one dictionary describing one token to match. The token attributes to match on can be the token’s "text" or lowercase form "lower", but also lexical attributes like "is_punct" or linguistic features like "lemma" or "pos". You can find more details in the spaCy’s documentation on rule-based matching.

2. String matches: If the pattern value is a string, it will be used for exact string matching. While {"lower": "berlin"} matches “Berlin”, “berlin” and so on, "Berlin" will only match “Berlin”. The advantage of string patterns is that you don’t have to worry about the tokenization and whether the patterns describe the correct tokens. They also make it easy to re-use existing word lists and dictionaries.

Manual annotation with suggestions from a model

Using a model to suggest entities is a great way to bootstrap training data for named entity recognition. For example, most general-purpose models were trained on large corpora of news and web text, annotated with at least a few generic entity types. Even if what you need is slightly different, you can still take advantage of what the model already predicts and let it highlight suggestions for you.

Let’s say you want to train a model for financial news with labels for person names, organizations, monetary amounts and ticker symbols. This is a very achievable task for named entity recognition. spaCy’s English models already predict PERSON, ORG and MONEY, so you can correct its suggestions for these labels and add annotations for your new TICKER label.

Instead of labelling all texts and entity types from scratch, you only need to fill in the blanks and fix incorrect predictions. If the model is decent enough, this can save you a lot of time. The workflow is also very helpful if you’ve already trained a custom model and want to get a better feeling for its predictions and perform error analysis.

Binary annotation with active learning and a model in the loop

The binary ner.teach workflow implements uncertainty sampling with beam search: for each example, the annotation model gets a number of analyses and asks you to accept or reject the entity analyses it’s most uncertain about. Based on your decisions, the model is updated in the loop and guided towards better predictions. Even if you don’t update with the full gold-standard annotation, confirming correct analyses and eliminating “dead ends” can very quickly move the model towards the right answer, since all token-based tagging decisions influence each other. Collecting binary feedback is also very fast and allows the annotator to focus on one concept and decision at a time, reducing the potential for human error and inconsistent data.

The following command starts the server and will make suggestions for the entity type ORG. The progress indicator in the sidebar shows an estimate of how much you still need to annotate until there’s nothing left to learn – or, phrased differently, an estimate of when the loss is going to hit zero.

If the suggested entity is fully correct, you can hit accept. If it’s entirely or partially wrong, you can hit reject. It’s important to only accept suggestions that are fully correct. As of v1.11, the ner.teach recipe will also ask you about examples containing no entities at all, which can improve overall accuracy of your model. So if you see an example with no highlighted suggestions, you can accept it if the text contains no entities, or reject it if it does contain entities of the labels you’re annotating.

The ner.teach recipe also supports match patterns via the --patterns argument. Pattern matches will be mixed in with the suggestions from the model and can help the model get over the “cold start” and make sure it starts off with enough positive examples to make meaningful suggestions. This can be used for entity types with very poor performance or even to add new entity types to an existing model.

prodigy train ./batch-trained-model --ner ner_dataset --base-model en_core_web_sm
prodigy ner.teach ner_dataset ./batch-trained-model ./data.jsonl --label PERSON

Efficient annotation for transformers like BERT New: 1.10

Transformer models like BERT typically use subword tokenization algorithms like WordPiece or Byte Pair Encoding (BPE) that are optimized for efficient embedding of large vocabularies and not necessarily linguistic definitions of what’s considered a “word”. If you’re creating training data for fine-tuning a transformer, you can use its tokenizer to preprocess your texts to make sure that the data you annotate is compatible with the transformer tokenization. It also makes annotation faster, because your selection can snap to token boundaries. The following recipe implementation uses Hugging Face’s easy-to-use tokenizers library under the hood and you can adjust it for your custom tokenization needs.

The recipe takes advantage of the "ws" property of each token to reflect whether tokens are followed by whitespace or not. This way, the text stays readable, even if words are split into several pieces. Setting --hide--wp--prefix hides the prefix added to word pieces by the tokenizer so the tokens are displayed as ["bi", "eber"] instead of ["bi", "##eber"]. You can also set --hide-special to hide tokens like [CLS] and [SEP] instead of just disabling them.

The data format produced by Prodigy retains the original texts and offsets produced by the tokenizer, as well as the encoded IDs as "tokenizer_id".

JSON data format
{
  "text": "Justin Bieber - agree 100000%",
  "tokens": [
    {"text": "[CLS]", "id": 0, "start": 0, "end": 0, "tokenizer_id": 101, "disabled": true, "ws": true},
    {"text": "justin", "id": 1, "start": 0, "end": 6, "tokenizer_id": 6796, "disabled": false, "ws": true},
    {"text": "bi", "id": 2, "start": 7, "end": 9, "tokenizer_id": 12170, "disabled": false, "ws": false},
    {"text": "eber", "id": 3, "start": 9, "end": 13, "tokenizer_id": 22669, "disabled": false, "ws": true},
    {"text": "-", "id": 4, "start": 14, "end": 15, "tokenizer_id": 1011, "disabled": false, "ws": true},
    {"text": "agree", "id": 5, "start": 16, "end": 21, "tokenizer_id": 5993, "disabled": false, "ws": true},
    {"text": "1000", "id": 6, "start": 22, "end": 26, "tokenizer_id": 6694, "disabled": false, "ws": false},
    {"text": "00", "id": 7, "start": 26, "end": 28, "tokenizer_id": 8889, "disabled": false, "ws": false},
    {"text": "%", "id": 8, "start": 28, "end": 29, "tokenizer_id": 1003, "disabled": false, "ws": true},
    {"text": "[SEP]", "id": 9, "start": 0, "end": 0, "tokenizer_id": 102, "disabled": true, "ws": true}
  ],
  "spans": [
     {"start": 0, "end": 13, "token_start": 1, "token_end": 3, "label": "PERSON"}
  ]
}

Working with longer texts

You probably noticed that most of the examples on this page show short texts like sentences or paragraphs. For NER annotation, there’s often no benefit in annotating long documents at once, especially if you’re planning on training a model on the data. Annotating with a model in the loop is also much faster if the texts aren’t too long, which is why recipes like ner.teach and ner.correct split sentences by default. NER model implementations also typically use a narrow contextual window of a few tokens on either side. If a human annotator can’t make a decision based on the local context, the model will struggle to learn from the data.

That said, there are always exceptions, and if you’re using the ner.manual workflow with whole documents, you can customize the UI theme to fit more text on the screen. For example:

prodigy.json (excerpt)
{
    "custom_theme": {"cardMaxWidth": "95%", "smallText": 16}
}

Prodigy will show you a little ↵ symbol to mark newlines, so you know where they are in your raw text. Tokens that only contain newlines are also unselectable by default, so you can’t accidentally include them in your annotated spans and confuse your model this way. See the ner_manual docs for more details about newline tokens.

Splitting long documents into pages New: 1.17

Alternatively, you can also use pages to present your longer documents split over multiple sections within the same annotation card. This can help reduce cognitive load for the annotator, while also keeping related content together. Pages can be provided as a list of "pages" following Prodigy’s JSON task format, or generated automatically from your data using the pages loader. For more details and examples, see the documetation on custom pages.

Using a custom model

You don’t need to use spaCy to let a model highlight suggestions for you. Under the hood, the concept is pretty straightforward: if you stream in examples with pre-defined "spans", Prodigy will accept and pre-highlight them. This means you can either stream in pre-labelled data, or write a custom recipe that uses your model to add "spans" to the examples.

Expected format
{
  "text": "Apple updates its analytics service with new metrics",
  "spans": [{"start": 0, "end": 5, "label": "ORG"}]
}

For inspiration, see the ner_make_gold.py example recipe script that shows how to add spaCy entities to the stream of incoming examples. For a custom model, the code could look like this:

Step 1: Write the stream generator
def add_entities_to_stream(stream):
   custom_model = load_your_custom_model()
   for eg in stream:
      ents = custom_model(eg["text"])
      eg["spans"] = [{"start": start, "end": end, "label"} for start, end, label in ents]
      yield eg

If you want to extract and add the entities at runtime, you can write a custom recipe that loads the raw data, uses your custom model to add "spans" to the stream, pre-tokenizes the text and then renders it all using the ner_manual interface.

Step 2: Putting it all together in a recipe
import prodigy
from prodigy.components.stream import get_stream
from prodigy.components.preprocess import add_tokens
import spacy

@prodigy.recipe("custom-ner")
def custom_ner_recipe(dataset, source):
    stream = get_stream(source)                     # load the data
    stream = add_entities_to_stream(stream)         # add custom entities
    stream = add_tokens(spacy.blank("en"), stream)  # add "tokens" to stream

    return {
        "dataset": dataset,          # dataset to save annotations to
        "stream": stream,            # the incoming stream of examples
        "view_id": "ner_manual",     # annotation interface to use
        "labels": ["PERSON", "ORG"]  # labels to annotate
    }

from spacy.tokens import Doc
from spacy.vocab import Vocab
from spacy.training import biluo_tags_to_offsets

doc = Doc(Vocab(), words=["I", "like", "New", "York"])
tags = ["O", "O", "B-LOC", "L-LOC"]
offsets = biluo_tags_to_offsets(doc, tags)  # [(7, 15, 'LOC')]

Active learning with a custom model

If you want to update your model in the loop as you annotate, make sure you pick an implementation that supports updates in small batches and that’s sensitive enough to small updates (since you want your annotations to have an effect).

Step 1: Use the model to predict and score entities
def predict(stream):
    for eg in stream:
        predictions = your_model(eg["text"])
        for score, start, end, label in predictions:
            example = copy.deepcopy(eg)
            example["spans"] = [{"start": start, "end": end, "label": label}]
            yield (score, example)

On their own, the scores and examples aren’t that interesting yet – you typically want to use the scores to only select the most relevant examples for annotation. In the built-in ner.teach recipe, we’re using beam search to get a number of different possible analyses and then use the prefer_uncertain sorter that takes the stream of (score, example) tuples and yields example dicts with scores closest to 0.5.

Step 2: Sort the stream by score
from prodigy.components.sorters import prefer_uncertain

stream = predict(stream)
stream = prefer_uncertain(stream)

Step 3: Update the model with answers
def update(answers):
    accepted = [eg for eg in answers if eg["answer"] == "accept"]
    rejected = [eg for eg in answers if eg["answer"] == "reject"]
    update_your_model(accepted, rejected)

By default, Prodigy streams are generators and Prodigy will only ever ask for the next batch from the stream. So as you annotate and update the model, future batches will receive scores from your updated model in the loop. For a simplified example of that loop, check out the textcat_custom_model.py recipe script. It shows a text classification use case, but the principle is very similar and is illustrated using a dummy model that “predicts” random numbers.

Step 4: Putting it all together in a recipe
import prodigy
from prodigy.components.stream import get_stream
from prodigy.components.sorters import prefer_uncertain

@prodigy.recipe("custom-ner")
def custom_ner_recipe(dataset, source):
    stream = get_stream(source)         # load the data
    stream = predict(stream)            # call custom predict function
    stream = prefer_uncertain(stream)   # sort to prefer uncertain scores

    return {
        "dataset": dataset,  # dataset to save annotations to
        "stream": stream,    # the incoming stream of examples
        "update": update,    # the update callback
        "view_id": "ner"     # annotation interface to use
    }

Training NER models

Once you’ve labelled some data with Prodigy, you can start your training experiments. If you’ve collected annotations from different sources or multiple annotators, it’s often a good idea to use the review recipe to resolve any conflicts and double-check the data. It’s also recommended to create a separate, dedicated evaluation set that you can compare different approaches against.

1. Train a spaCy pipeline using Prodigy’s CLI. The train recipe is a wrapper around spaCy’s training API and optimized for training straight from Prodigy datasets and quick experiments. It reads from a dataset, holds back data for evaluation and outputs nicely-formatted results. This workflow is the best choice if you just want to get going or quickly check if you’re “on the right track” and your model is learning things.

2. Train a model with spaCy directly. Once you’re getting more serious, it often makes sense to train your model directly with the library you’re using – e.g. spaCy. This gives you more control over the training process and hyperparameters, and lets you train all model components at once. The data-to-spacy command lets you convert Prodigy datasets to spaCy’s format and auto-generates a config to use with the spacy train command. It’s recommended to use the review recipe on the different annotation types first to resolve conflicts properly. To check if your data is valid and contains no issues, you can run spaCy’s debug-data command.

3. Train a model with any other implementation or framework. The db-out exports annotations in a straightforward JSONL format. Entity spans are provided as a list of "spans" with their respective start and end character offsets, and the entity label. This should make it easy to convert and use it to train any model. If you annotated using a manual interface, the data will also include a "tokens" property so you can restore the tokenization.

import spacy
from spacy.gold import biluo_tags_from_offsets

text = "I like New York City"
spans = [{"start": 7, "end": 20, "label": "LOC"}]
nlp = spacy.blank("en")
doc = nlp(text)

offsets = [(span["start"], span["end"], span["label"]) for span in spans]
biluo_tags = biluo_tags_from_offsets(doc, offsets)
# ['O', 'O', 'B-LOC', 'I-LOC', 'L-LOC']
doc.ents = [doc.char_span(start, end, label) for start, end, label in offsets]
iob_tags = [f"{t.ent_iob_}-{t.ent_type_}" if t.ent_iob_ else "O" for t in doc]
# ['O', 'O', 'B-LOC', 'I-LOC', 'I-LOC']

Improving accuracy and using transfer learning

After you’ve done your initial annotations and run your first training process, you’ll usually be interested in improving the accuracy of your model. After all, you’re unlikely to get to 100% accuracy right away – and if you ever do, that’s probably a sign that something’s wrong! Here are the steps we generally recommend to improve your initial accuracies:

1. Check if you just need more data. The train-curve recipe is a helpful utility for checking if your model would improve with more data. It takes the same arguments as the train and will train with different samples of the data – for instance, 25%, 50%, 75% and 100%. As a rule of thumb, if accuracy improves within the last segment, it could indicate that more annotations of the same type will improve the model.

2. Initialize with word vectors. Using pretrained word embeddings to initialize your model is easy and can make a big difference. If you’re using spaCy, try using the en_core_web_lg model as the base model. If you’re working with domain-specific texts, you can train your own vectors and create a base model with them.

3. Initialize with pretrained tok2vec weights or transformer embeddings. Transfer learning has proven to be a very effective way to “share” knowledge between tasks, by initializing models with pretrained representations. Transfer learning is quite easy to use, but there are a few steps in the workflow. For an end-to-end example, check out this project with code, data and downloadable weights. Compared to just training with word vectors, an NER model trained on the same data achieved +8.7% accuracy. You can use spaCy’s config generator or init config command to generate a config for a transformer-based pipeline, and data-to-spacy to export your annotations for training.