API

Annotation interfaces

The web app lets you annotate a variety of different formats, including plain text, named entities, categorization, images and A/B comparisons, as well as raw HTML. Prodigy expects annotation tasks to follow a simple JSON-style format. This format is also used to communicate tasks between REST API and web application, and will be used when exporting annotations via the db-out command. Annotated tasks contain an additional "answer" key mapping to either "accept", "reject" or "ignore".

text, htmlAnnotate plain text or HTML.
ner, ner_manualAnnotate and correct named entities.
pos, pos_manualAnnotate and correct part-of-speech tags.
depAnnotate syntactic dependencies and semantic relations.
classificationAnnotate labelled text or images.
image, image_manualCreate bounding boxes and image segments.
choiceSelect one or more answers.
reviewReview and merge existing annotations by multiple annotators.
diff, compareCompare two texts with a visual diff.
text_inputCollect free-form text input.
blocksCombine different interfaces and custom content.

text Annotate plain textbinary

This live demo requires JavaScript to be enabled.
JSON task format{
  "text": "Nintendo Switch",
}

The text property of an annotation task will always be rendered as plain text. To add markup like line breaks or simple formatting, use the html key instead. Prodigy wants you to explicitly choose to use “HTML mode” to avoid stray HTML tags (which can influence the model’s predictions) from being rendered or hidden – for example, if you’re working with raw, uncleaned data. If you’re using one of Prodigy’s default recipes with a model in the loop, keep in mind that the text of an annotation task is used to update the model.


ner Annotate named entitiesbinary

The ner interface renders one or more entity spans in a text using their start and end character offsets and an optional label. If you’re creating NER annotation tasks manually and need to render multiple entities per task, make sure to provide them in order.

This live demo requires JavaScript to be enabled.
JSON task format{
  "text": "Apple updates its analytics service with new metrics",
  "spans": [{ "start": 0, "end": 5, "label": "ORG" }]
}

ner_manual Annotate and correct named entitiesmanual

The ner_manual interface allows highlighting spans of text based on token boundaries, and will use a model to tokenize the text and add an additional "tokens" property to the annotation task. This allows the application to resolve the highlighted spans back to the respective token boundaries. Selected entities will be added to the task’s "spans". Individual tokens can also set "disabled": true to make them unselectable (and make spans including those tokens invalid).

This live demo requires JavaScript to be enabled.
JSON task format{
  "text": "First look at the new MacBook Pro",
  "spans": [
    {"start": 22, "end": 33, "label": "PRODUCT", "token_start": 5, "token_end": 6}
  ],
  "tokens": [
    {"text": "First", "start": 0, "end": 5, "id": 0},
    {"text": "look", "start": 6, "end": 10, "id": 1},
    {"text": "at", "start": 11, "end": 13, "id": 2},
    {"text": "the", "start": 14, "end": 17, "id": 3},
    {"text": "new", "start": 18, "end": 21, "id": 4},
    {"text": "MacBook", "start": 22, "end": 29, "id": 5},
    {"text": "Pro", "start": 30, "end": 33, "id": 6}
  ]
}

A newline only produces a line break and is otherwise invisible. So if your input data contains many newlines, they can be easy to miss during annotation. To your model however, \n is just a unicode character. If you’re labelling entities and accidentally include newlines in some of them and not others, you’ll end up with inconsistent data and potentially very confusing and bad results. To prevent this, Prodigy will show you a for each newline in the data (in addition to rendering the newline itself). The tab character \t will be replaced by , by the way, for the same reason.

As of v1.9, tokens containing only newlines (or only newlines and whitespace) are unselectable by default, so you can’t include them in spans you highlight. To disable this behavior, you can set "allow_newline_highlight": true in your prodigy.json.

When you’re highlighting spans in the manual interface, you’re still annotating raw text and are creating spans that map to character offsets within that raw text. If you pass in "<strong>hello</strong>", there’s no clear solution for how this should be handled. How should it be tokenized, and what are you really labelling here? The underlying markup or just the text, and what should the character offsets point to? And how should other markup be handled, e.g. images or complex, nested tags?

Similarly, if you’re planning on training a model later on, that model will also get to see the raw text, including the markup – so if you are working with raw HTML (like web dumps), you usually always want to see the original raw text that the model will be learning from. Otherwise, the model might be seeing data or markup that you didn’t see during annotation, which is problematic.


pos Annotate part-of-speech tagsbinary

Under the hood, the POS tagging interface uses the same UI component as the ner interface. It also accepts and produces the same data format. The default theme ships with several customizable highlight colors for part-of-speech tags that make it easier to distinguish them.

This live demo requires JavaScript to be enabled.
JSON task format{
  "text": "First look at the new MacBook Pro",
  "spans": [{"start": 6, "end": 10, "label": "NOUN"}]
}

pos_manual Annotate and correct part-of-speech tagsmanual

Under the hood, the manual POS tagging interface uses the same UI component as the ner_manual interface. It also accepts and produces the same data format. The default theme ships with several customizable highlight colors for part-of-speech tags that make it easier to distinguish them.

This live demo requires JavaScript to be enabled.
JSON task format{
  "text": "First look at the new MacBook Pro",
  "spans": [
    {"text": "First", "start": 0, "end": 5, "token_start": 0, "token_end": 0, "label": "ADJ"},
    {"text": "look", "start": 6, "end": 10, "token_start": 1, "token_end": 1, "label": "NOUN"},
    {"text": "new", "start": 18, "end": 21, "token_start": 4, "token_end": 4, "label": "ADJ"},
    {"text": "MacBook", "start": 22, "end": 29, "token_start": 5, "token_end": 5, "label": "PROPN"},
    {"text": "Pro", "start": 30, "end": 33, "token_start": 6, "token_end": 6, "label": "PROPN"}
  ],
  "tokens": [
    {"text": "First", "start": 0, "end": 5, "id": 0},
    {"text": "look", "start": 6, "end": 10, "id": 1},
    {"text": "at", "start": 11, "end": 13, "id": 2},
    {"text": "the", "start": 14, "end": 17, "id": 3},
    {"text": "new", "start": 18, "end": 21, "id": 4},
    {"text": "MacBook", "start": 22, "end": 29, "id": 5},
    {"text": "Pro", "start": 30, "end": 33, "id": 6}
  ]
}

dep Annotate syntactic dependencies and semantic relationsbinary

Dependencies are specified as a list of "arcs" with a "head" and a "child" token index, indicating the direction of the arrow. Since the arcs refer to the token index, the task also requires a "tokens" property. This is usually taken care of in the recipes using the add_tokens pre-processor.

This live demo requires JavaScript to be enabled.
JSON task format{
  "text": "First look at the new MacBook Pro",
  "arcs": [{"head": 2, "child": 5, "label": "pobj"}],
  "tokens": [
    {"text": "First", "start": 0, "end": 5, "id": 0},
    {"text": "look", "start": 6, "end": 10, "id": 1},
    {"text": "at", "start": 11, "end": 13, "id": 2},
    {"text": "the", "start": 14, "end": 17, "id": 3},
    {"text": "new", "start": 18, "end": 21, "id": 4},
    {"text": "MacBook", "start": 22, "end": 29, "id": 5}
  ]
}

classification Annotate labelled text or imagesbinary

The classification interface lets you render content (text, text with spans, image or HTML) with a prominent label on top. The annotation decision you typically collect with this interface is whether the label applies to the content or not.

This live demo requires JavaScript to be enabled.
JSON task format (text){
  "text": "Apple updates its analytics service with new metrics",
  "label": "TECHNOLOGY"
}
This live demo requires JavaScript to be enabled.
JSON task format (text with spans){
  "text": "Apple updates its analytics service with new metrics",
  "label": "CORRECT",
  "spans": [{ "start": 0, "end": 5, "label": "ORG" }]
}
This live demo requires JavaScript to be enabled.
JSON task format (image){
  "image": "https://images.unsplash.com/photo-1536521642388-441263f88a61?w=400",
  "label": "FOOD"
}

image Annotate images, bounding boxes and image segmentsbinary

Images can be provided as file paths, URLs or base64-encoded data URIs – basically, anything that can be rendered as an image. When using local paths, keep in mind that modern browsers typically block images from local paths for security reasons. So you can either host the images with a local web server or in an S3 bucket, or use the fetch_images preprocessor to convert all images from local paths and URIs to base64-encoded data URIs.

This live demo requires JavaScript to be enabled.
JSON task format (image URL){
  "image": "https://images.unsplash.com/photo-1554415707-6e8cfc93fe23?w=400"
}
JSON task format (base64 data URI){
  "image": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWFnZVJlYWR5ccllPAAAAqdJREFUeNrMl12LTVEcxs85TjPMKBkmeWkaOWgkKa9z4T1ulCtlcDG5UFO4QG58BD4ApcSESe7kQilcaGQMbpzMSJozNElMQ5IhM36P/mq3O3ufvdbZu6z6tTp77bXWs/b/bZ1c7n9si9pKcz3n5cFpTiHi+U5P7SthRhoCVrmexNoaaEpDwFaY5fkFFtYlgJM30q2FNg8B7a7zqn2BTpgHpxBT8BDQ7jKhaLZugGY7+TnIQze8Zvwm/Uf4NTb6ZjomAv52PiYomOojcBHW25hMcR6uwu4EzqUDLIblLgLyoVNokz7YB1NwQqI4+VSCHNBBV4ZB3t/o5QNM/E53Gn7DbbiQZHNrW+xAJcTMrMcJR2FcAuJsHpO85pgp/ASw6U85HzxySMEL6PYG1uyqNxE9gXcJN281R50deHyW50ehwckJA4vK6++FIqVokSFH1aYrYDschJaI9YfgOjy1A8m0X2ESZN7pKAEtmGLcElGrhekSWA07oMM2LSb80j9gAj7BMzPvK0VN3qPkNlqtOAz7Lf7jmiLqPtyCB/A2GFnOAkJiltJdgl0RrwzCcRiIiqh8wo1U40ssMhzxRZQz9oSGrsAxyy3OURBuKs2bqw2wwaTVj+AJx+Bkrc1dBMgRt8WMP7Tc8a/1svlEPXkg3FTqNunOF/EV5GjXQoJyaQrYAMus3Ea1uwGvL6ciQDUeVF4PWRI6w+/miPtiRXcG+AafvcqxbarkMt8umF0W78GUqsVvwB14rt8KMYsUFbIP/F7nLSAkRl/oAFwGldgRE/S4WlzzvmxfYaw78ZUsbtAyVh8Ld9rlpIdn/TFTymaKXCoCAm3A7PuixnsjWQl4acmllnOpwLzPQoBO1m/xHteUqr9kIUCL9iZ4r2J5IJN/zE1ZrPtHgAEAAuvBpEMJ1j8AAAAASUVORK5CYII="
}

Image tasks can also define a list of "spans", describing the boxes to draw on the image. Each span can define a list of "points", consisting of [x, y] coordinate tuples of the bounding box corners. While bounding boxes are usually squares, Prodigy also supports polygon shapes. The coordinates should be measured in pixels, relative to the original image. If a "label" is specified, it will be rendered with the bounding box. You can also define an optional "color". If none is set, Prodigy will pick one for you.

This live demo requires JavaScript to be enabled.
JSON task format (with boxes){
  "image": "https://images.unsplash.com/photo-1554415707-6e8cfc93fe23?w=400",
  "spans": [{"points": [[155, 15], [305, 15], [305, 160], [155, 160]], "label": "LAPTOP"}]
}

image_manual Create bounding boxes and image segmentsmanual

The manual image interface lets you draw bounding boxes and polygon shapes on the image. It’s optimized for fast object annotation, but not so much high-precision image masking (for which we think specialized tools are a much better fit). You can click and drag or click and release to draw boxes. Polygon shapes can also be closed by double-clicking when adding the last point, similar to closing a shape in Photoshop or Illustrator. Clicking on the label will select a shape so you can change the label or delete it. The interface also supports pre-populating the "spans" to pre-highlight shapes.

This live demo requires JavaScript to be enabled.
JSON task format{
  "image": "https://images.unsplash.com/photo-1554415707-6e8cfc93fe23?w=400",
  "width": 400,
  "height": 267
}
This live demo requires JavaScript to be enabled.
JSON task format (with spans){
    "image": "https://images.unsplash.com/photo-1415594445260-63e18261587e?w=1080",
    "width": 1080,
    "height": 720,
    "spans": [
        {"label": "CAR", "points": [[6,6],[296,3],[343,22],[396,133],[433,143],[431,194],[410,199],[432,269],[426,349],[433,471],[406,490],[386,511],[381,618],[360,670],[289,687],[264,665],[107,678],[3,677],[6,6]]},
        {"label": "CAR", "points": [[904,98],[876,140],[854,209],[838,222],[825,313],[825,354],[829,371],[825,458],[832,523],[857,533],[899,532],[912,512],[972,591],[975,666],[995,712],[1074,714],[1076,65],[1007,71],[935,79],[912,87],[904,98]]},
        {"label": "CAR", "points": [[753,33],[818,30],[1000,30],[1037,35],[1044,68],[933,80],[911,90],[879,138],[858,195],[852,213],[836,226],[825,307],[817,310],[814,375],[796,381],[750,375],[744,331],[716,328],[712,253],[705,213],[706,170],[730,158],[735,86],[742,48],[753,33]]},
        {"label": "CAR", "points": [[662,73],[694,66],[738,65],[731,158],[703,173],[708,220],[712,267],[680,267],[676,281],[640,278],[634,217],[634,145],[616,134],[616,127],[640,122],[662,73]]}
    ]
}

As of v1.8.4, Prodigy lets you move the selected bounding box using the keyboard shortcuts . This is especially useful to correct smaller positioning errors. Prodigy currently doesn’t support editing individual points and resizing boxes. If you want to edit a box, you’ll need to remove and redraw it. which often isn’t more work and lets us keep the interface and usage straightforward.


choice Select one or more answersmanual

The choice interface can render text, entities, images or HTML. Each option is structured like an individual annotation task and will receive a unique ID. You can also choose to assign the IDs yourself. A choice task can also contain other task properties – like a text, a label or spans. This content will be rendered above the options.

Note that selecting options via keyboard shortcuts like 1 and 2 only works for 9 or less options. To use multiple choice instead of single choice, you can set"choice_style": "multiple" in your Prodigy or recipe config. You can also set "choice_auto_accept": true to automatically accept a single-choice option when selecting it – without having to click the accept button.

This live demo requires JavaScript to be enabled.
This live demo requires JavaScript to be enabled.
JSON task format{
  "text": "Pick the odd one out.",
  "options": [
    {"id": "BANANA", "text": "🍌 banana"},
    {"id": "BROCCOLI", "text": "🥦 broccoli"},
    {"id": "TOMATO", "text": "🍅 tomato"}
  ]
}

When you annotate, an "accept" key is added to the parent task, containing the IDs of the selected options, as well as the "answer". This allows the annotator to also reject or ignore tasks, for example if they contain errors.

JSON task format (annotated){
  "text": "Pick the odd one out.",
  "options": [
    {"id": "BANANA", "text": "🍌 banana"},
    {"id": "BROCCOLI", "text": "🥦 broccoli"},
    {"id": "TOMATO", "text": "🍅 tomato"}
  ],
  "accept": ["BROCCOLI"],
  "answer": "accept"
}

An option can also contain a style key with an object of CSS properties in the camel-cased JavaScript format, mapping to their values:

JSON task format (with custom CSS){
  "options": [
    {"id": 1, "text": "Option 1", "style": {"background": "#ff6666"}},
    {"id": 2, "text": "Option 2", "style": {"fontWeight": "bold" }}
  ]
}
This live demo requires JavaScript to be enabled.

html Annotate HTML content

The HTML interface can render any HTML, including embeds. It’s especially useful for basic formatting and simple markup you can create programmatically. If you’re looking to build complex or interactive custom interfaces, you probably want to use the "html_template" config setting and add custom CSS and JavaScript.

This live demo requires JavaScript to be enabled.
JSON task format{
  "html": "<iframe width='400' height='225' src='https://www.youtube.com/embed/vKmrxnbjlfY'></iframe>"
}

review Review and merge existing annotations by multiple annotatorsNew: 1.8

The review interface currently supports all annotation modes except image_manual and compare. It will present one or more versions of a given task, e.g. annotated in different sessions by different users, display them with the session information and allow the reviewer to correct or override the decision. The data requires a "view_id" setting that defines the annotation UI to render the example with. The versions follow the same format as regular annotation tasks. They can specify a list of "sessions", as well as a boolean "default" key, marking that version as the default version to pre-populate the review UI with (if the interface is a manual interface).

This live demo requires JavaScript to be enabled.
JSON task format (binary annotations){
  "text": "Hello world",
  "label": "POSITIVE",
  "view_id": "classification",
  "versions": [
    {
      "text": "Hello world",
      "label": "POSITIVE",
      "answer": "accept",
      "sessions": ["dataset-user1", "dataset-user3", "dataset-user4"]
    },
    {
      "text": "Hello world",
      "label": "POSITIVE",
      "answer": "reject",
      "sessions": ["dataset-user2"]
    }
  ]
}
This live demo requires JavaScript to be enabled.
JSON task format (manual annotations){
  "text": "Hello Apple",
  "tokens": [{"text": "Hello", "start": 0, "end": 5, "id": 0}, {"text": "Apple", "start": 6, "end": 11, "id": 1}],
  "answer": "accept",
  "view_id": "ner_manual",
  "versions": [
    {
      "text": "Hello Apple",
      "tokens": [{"text": "Hello", "start": 0, "end": 5, "id": 0}, {"text": "Apple", "start": 6, "end": 11, "id": 1}],
      "spans": [{"start": 6, "end": 11, "label": "ORG", "token_start": 1, "token_end": 1}],
      "answer": "accept",
      "sessions": ["dataset-user1", "dataset-user3", "dataset-user4"],
      "default": true
    },
    {
      "text": "Hello Apple",
      "tokens": [{"text": "Hello", "start": 0, "end": 5, "id": 0}, {"text": "Apple", "start": 6, "end": 11, "id": 1}],
      "spans": [],
      "answer": "accept",
      "sessions": ["dataset-user2"],
      "default": false
    }
  ]
}

diff Compare two texts with a visual diff

The diff view is especially useful if you’re dealing with very subtle comparisons, e.g. when evaluating spelling correction. The config setting "diff_style" can be set to "chars", "words" or "sentences" to customize how the diff is displayed. If you prefer the suggested changes, you can hit accept, which corresponds to the "accept" object in the data. In a real-world evaluation setting, you would typically randomize which text gets shown as "accept" and "reject" and then store that mapping with the task, so you can resolve it later.

This live demo requires JavaScript to be enabled.
JSON task format{
  "accept": {
      "text": "Cyber researchers have linked the vulnerability exploited by the latest ransomware to “WannaCry”. Both versions of malicious software rely on weaknesses discovered by the National Security Agency years ago, Kaspersky said."
  },
  "reject": {
      "text": "Cyber researchers linked the vulnerability exploited by the latest ransomware to 'Wanna Cry'. Both versions of malicious software rely on weaknessses, discovered by the National Security Agency years ago, Kaspersky said"
  }
}

compare Compare two pieces of content

The compare interface, typically used via the compare recipe, supports various types of content like text, HTML and images. It can receive an optional input containing the original example, plus two examples to collect feedback on. This format is normally used to compare two different outputs, and requires you to feed in data from two different sources. The data is associated via its IDs. To ensure unbiased annotation, Prodigy randomly decides which example to return as the assumed correct answer, e.g. "accept". The mapping lets you resolve those back to the original sources later on (A or B).

This live demo requires JavaScript to be enabled.
JSON task format{
  "id": 1,
  "input": {"text": "NLP"},
  "accept": {"text": "Natural Language Processing"},
  "reject": {"text": "Neuro-Linguistic Programming"},
  "mapping": {"A": "accept", "B": "reject"}
}

text_input Collect free-form text inputNew: 1.9

The text input interface lets you collect free-form text input. The text is added to the annotated task with a given key. You can also customize the placeholder text, add an optional label to the field and make the input field a text area with multiple rows. The text input interface is best used as a block in the blocks interface, for example to ask the annotator to translate a given text or leave feedback about a manual NER annotation task.

This live demo requires JavaScript to be enabled.
JSON task format{
  "field_id": "user_input",
  "field_label": "User input field",
  "field_placeholder": "Type here...",
  "field_rows": 1,
  "field_autofocus": false
}

The "field_id" property lets you define a custom key that the collected text is stored under. You can also pre-populate the field when you stream in the data to set a default value. This can be very useful for tasks that need the annotator to rewrite a text or perform other manual corrections.

JSON task format (annotated){
  "field_id": "user_input",
  "user_input": "This is some text typed by the user",
  "answer": "accept"
}

blocks Combine different interfaces and custom contentNew: 1.9

The blocks interface lets you combine different interfaces via the custom recipe configuration. A list of "blocks" tells it which interfaces to use in which order. The incoming data needs to match the format expected by the interfaces used in the blocks. For instance, to render a ner_manual block, the data needs to contain "text" and "tokens". For more details and examples, see the documentation on custom blocks.

This live demo requires JavaScript to be enabled.
Recipe config{
  "labels": ["PERSON", "ORG", "PRODUCT"],
  "blocks": [
    {"view_id": "ner_manual"},
    {"view_id": "text_input"},
    {"view_id": "html"},
  ]
}
Example JSON task{
  "text": "First look at the new MacBook Pro",
  "spans": [
    {"start": 22, "end": 33, "label": "PRODUCT", "token_start": 5, "token_end": 6}
  ],
  "tokens": [
    {"text": "First", "start": 0, "end": 5, "id": 0},
    {"text": "look", "start": 6, "end": 10, "id": 1},
    {"text": "at", "start": 11, "end": 13, "id": 2},
    {"text": "the", "start": 14, "end": 17, "id": 3},
    {"text": "new", "start": 18, "end": 21, "id": 4},
    {"text": "MacBook", "start": 22, "end": 29, "id": 5},
    {"text": "Pro", "start": 30, "end": 33, "id": 6}
  ],
  "field_id": "user_input",
  "field_rows": 5,
  "html": "<iframe width='100%' height='150' scrolling='no' frameborder='no' src='https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/tracks/174136820&color=%23ff5500&auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&show_teaser=true&visual=true'></iframe>"
}