Web Application

When humans interact with machines, their experience is what decides about the success of the interaction. Prodigy helps you collect annotations faster and more efficiently by focusing on one task at a time, and reducing each annotation to a simple binary decision. This means you'll collect more user actions overall, giving you lots of smaller pieces to learn from, and a much tighter feedback loop between the human and the model.

Screenshot of the Prodigy web app and its components

The web app includes a range of built-in annotation interfaces, key bindings and mobile swipe gestures and fully customisable UI themes. When you annotate a task, the decision – accept, reject or ignore – is added to the task as the answer key. The ten most recent decisions are shown in the sidebar and remain editable, either by clicking "undo", or by selecting them from the history. Annotated tasks are sent back to the REST API in batches. To save the current progress manually, you can always hit the save button, or press command+s or ctrl+s.

Annotation interfaces

The web app lets you annotate a variety of different formats, including plain text, named entities, categorisation, images and A/B comparisons, as well as raw HTML.

Text
Annotate plain text
text
MacBook
Named Entity Recognition
Annotate and correct named entities
ner ner_manual
First look at the new MacBook Pro product
Part-of-speech Tagging
Annotate and correct part-of-speech tags
pos pos_manual
First adj look noun at adp the det new adj MacBook propn Pro propn
Dependency Parsing
Annotate syntactic dependencies and semantic relations
dep
First look pobjat the new MacBook
Classification
Annotate labelled text or images
classification
Technology
First look at the new MacBook Pro
Image
Annotate images, bounding boxes and image segments
image image_manual
 laptop 
Choice
Select one or more answers or pick the odd one out
choice
Compare
Compare two annotations
compare
Technology
Technologie
Technik
Diff
Compare texts with a visual diff
diff
Is it time to swap your Mac for a Windows computerlaptop?
HTML
Annotate HTML content
html
#Quarter 1Quarter 2
113,432.0012,401.00
25011.001290.00
320.5099.00

Key bindings and swipe gestures

For more convenient annotation, you can use the following key bindings. You can also enable swipe gestures on touch devices, which will let you swipe left for accept and swipe right for reject.

ActionDescriptionKey options
acceptAccept an annotation. a
rejectReject an annotation. x
ignoreIgnore an annotation. space
undoUndo the last annotation decision. backspace del
saveSave the current buffer of annotations. command+s ctrl+s
flagFlag an example to bookmark it for later. f
helpShow annotation instructions, if available. h
choiceSelect a label in a manual interface or an option in a choice interface. 0 - 9

Statistics and progress

The sidebar helps you keep an overview of your annotation project and includes a progress widget showing the project's annotation count, progress and optional detailed statistics for the accept, reject and ignore decisions. To re-add an already annotated task to the front of the queue and change your decision, simply click on it in the history, or use the undo button or the backspace key.

Progress bar with annotation counts
List of recently annotated tasks

Flagging tasks

Flagging a task lets you to bookmark it for later, independent of the annotation decisions accept, reject and ignore. This can be useful to keep a record of seemingly strange predictions of your model, to investigate later on, or to save interesting examples you come across. Since flagging is mostly developer functionality, it's disabled by default and requires the "show_flag" setting. Flagged tasks will receive a "flagged": true entry, and can be exported separately by setting --flagged-only on db-out.

This is a weird task you want to bookmark for later

Annotation instructions

If you need to provide more detailed annotation instructions, you can use the "instructions" config setting to load in a text or HTML file. The instructions will then be displayed in a modal, accessible via the help icon at the top, or by pressing h.

Themes

Prodigy comes with three built-in theme options. To change the theme, simply edit the theme setting in your configuration file and restart the web app.

You can also overwrite a range of colours and other layout properties under the custom_theme setting to customise the look and feel of the application.

SettingDescriptionDefault
acceptColor to visualise "accept" action.#4fd364
rejectColor to visualise "reject" action.#f74c4a
ignoreColor to visualise "ignore" action.#b9b9b9
undoColor to visualise "undo" action.#b9b9b9
bgCardBackground of annotation card.#ffffff
bgPagePage background.#f6f6f6
bgSidebarGeneral sidebar background.#54606e
bgSidebarDarkDarker shade of sidebar color for header and accents.#384451
bgHighlightBackground of highlighted text, e.g. entities.#ffe184
bgLowlightBackground of subtly highlighted text, e.g. spans outside entities.#d9d9d9
bgCardTitleBackground color of card title, e.g. text category.#583fcf
bgProgressColor of progress bar.#ffffff
bgButtonDefault annotation button background (disabled).#b9b9b9
bgMetaBackground of card meta text.#f6f6f6
colorTextGeneral text color.#444444
colorMetaText color of card meta text.#939393
colorMessageText color of loading or error message if no card is displayed.#6e6e6e
colorSidebarText color in sidebar.#ffffff
colorSidebarHeadlineText color of sidebar headlines.#ffffff
colorSidebarLabelText color of of sidebar subheadlines, e.g. "Dataset".#b9b9b9
colorButtonIcon color of annotation buttons.#ffffff
colorHighlightLabelText color of label on highlighted text, e.g. entity label.#583fcf
colorCardTitleText color of card title, e.g. category.#ffffff
cardBorderBorder around card.1px solid #ddd
cardShadowBox shadow around card.1px 1px 5px #ddd
sidebarBorderRight border on sidebar.none
labelsCustom highlight colors for labels, keyed by label name.see below

You can also customise a theme's fonts. The primary font is used for most text, while the secondary font is used for headlines and labels.

SettingDefault
fontPrimary"Lato", "Trebuchet MS", Roboto, Helvetica, Arial, sans-serif
fontSecondary"Roboto Condensed", "Arial Narrow", sans-serif

Using the "labels" setting, a theme can specify custom highlight colors for entity labels, part-of-speech tags etc. The colors will be used in both the manual and non-manual interfaces. If no label color is set, the default bgHighlight is used.

LabelTypeDefault
all labelsentity labels, part-of-speech tags, other spans#ffe184
Olabel for spans outside of an entity#d9d9d9
VERBpart-of-speech tag#ffa9b8
NOUNpart-of-speech tag#d1caff
PROPNpart-of-speech tag#b4e8fc
ADJpart-of-speech tag#d3fe7d
ADVpart-of-speech tag#9effd6
CONJ, CCONJpart-of-speech tag#ffa973
PUNCTpart-of-speech tag#dddddd

Custom interfaces with HTML templates

The html interface lets you render any plain HTML content, specified as the "html" key in the annotation task. If you don't want to include the full markup with every task, you can specify a html_template in your global or the project's configuration file.

Annotation task

{ "title": "This is a title", "image": "https://images.unsplash.com/photo-1472148439583-1f4cf81b80e0?ixlib=rb-0.3.5&q=80&fm=jpg&crop=entropy&cs=tinysrgb&w=400&fit=max&s=21dba460262331eb682c38680f7f1135", "data": { "label": "Some label", "value": "This is a value" }, "meta": { "source": "Unsplash", "by": "Peter Nguyen", "url": "https://unsplash.com/@peterng1618" } }

Prodigy uses Mustache to render the templates, and makes all task properties are available as variables. With HTML templates, you can visualise complex, nested data without having to convert your input to match Prodigy's format.

HTML template

<img src="{{image}}" /> <h2>{{title}}</h2> <strong>{{data.label}}</strong> <span style="background: #ffe184">{{data.value}}</span>
Photo of a city skyline

This is a title

Some label This is a value
source: Unsplash by: Peter Nguyen url: unsplash.com/@peterng1618
scikit