Prepares a local disk cache to speed up inference for segment.image.manual. This can cause a huge speedup if you’re running on a non-GPU device.

prodigy
segment.fill-cache
source
--loader
checkpoint
--model-type
--cache

Prodigy-ANN

Sometimes you may want to query your examples to find a relevant subset for annotation. A modern method for doing this is to use numeric vectors to represent text and you can use approximate neighest neighbor (ANN) techniques to fetch relevant examples. The goal is to spend more time looking at examples that matter, like examples similar to items that the model gets wrong. Curating these examples first might be a pragmatic method to steer the model in the right direction.

If you’re interested to see a quick demo for Prodigy-ANN applied to a text dataset, you may appreciate this Prodigy short on Youtube.

To use this plugin, you’ll need to install it first.

Install prodigy-annpip install "prodigy-ann @ git+https://github.com/explosion/prodigy-ann"

As a first step for this approach you’ll first need to generate an index with vector representations of your text. To encode the text this library uses sentence-transformers and it uses hnswlib as an index for these vectors.

To index your documents, you can run the ann.text.index recipe.

prodigy
ann.text.index
examples.jsonl
examples.index

indexing: 100%|███████████████████████████| 2210/2210 [00:09<00:00, 243.64it/s]

Once it is indexed you can use text queries find and curate interesting subsets. A general method to prepare these subsets is to use ann.text.fetch. This will fetch a subset of vectors that are close in vector space and save the associated examples on disk. From there you can use any Prodigy recipe you like.

prodigy
ann.text.fetch
examples.jsonl
examples.index
subset.jsonl
--query "this is an outrage!"

More interfaces

As a convenience this plugin also provides the textcat.ann.manual, ner.ann.manual and spans.ann.manual so that you may query and annotate directly. These recipes have the same arguments as their native Prodigy textcat.manual, ner.manual and spans.manual counterparts but add a --query parameter so that you may pass your query.

Interactive Queries

Sometimes you may want to update the stream while you’re annotating. You can do that without restarting the server by using the --allow-reset flag when you’re starting the textcat.ann.manual, ner.ann.manual or spans.ann.manual recipes.

prodigy
textcat.ann.manual
examples.jsonl
examples.index
--query "new academic dataset"
--allow-reset

Here’s an example of what the experience might look like from the UI.

Retreiving Images

You can use these embedding retreival techniques for images too. Models like CLIP allow you to embed images and text in the same space, which means that you can query the images by using text.

The approach for images is very similar to the approach for text too. To get started you’ll first want to run an indexing recipe over a folder of images via the ann.image.index recipe.

prodigy
ann.image.index
path/to/image_folder
image.index

indexing: 100%|███████████████████████████| 210/210 [01:49<00:00]

Once the index is built, you can query it. You can choose to query it to prepare a .jsonl file to re-use later via the ann.image.fetch recipe.

prodigy
ann.image.fetch
path/to/image_folder
examples.index
out.jsonl
--query "laptops"
--remove-base64
--n 100

Alternatively the plugin also provides a wrapper around the familiar image.manual recipe. This will retreive the images before passing it on to the image_manual interface. This interface also allows you to reset the stream via the --allow-reset flag.

prodigy
image.ann.manual
annotated_laptops
path/to/image_folder
examples.index
--query "laptops"
--remove-base64
--n 100
--labels laptop,phone
--allow-reset

Here’s an example of what the experience might look like from the UI.

API

prodigy
ann.text.index
source
examples.index

prodigy
ann.text.fetch
source
index_path
out_path
--query
--n

prodigy
ann.image.index
source
examples.index

prodigy
ann.image.fetch
source
index_path
out_path
--query
--query
--remove-base64

Prodigy-Lunr

Instead of using semantic vectors with approximate nearest neighbors to find relevant subsets you can also resort to the “regular” search techniques. To accomodate these techniques we’ve added support for recipes that use lunr. These recipes are very similar to their ann.* counterparts but will rely on string matching techniques to retreive relevant examples.

To use this plugin, you’ll need to install it first.

Install prodigy-lunrpip install "prodigy-lunr @ git+https://github.com/explosion/prodigy-lunr"

To index your documents, you can run the ann.text.index recipe. This will generate an index and serialize it to disk by writing it into a gzipped json file.

prodigy
lunr.text.index
examples.jsonl
index.gz.json

indexing: 100%|███████████████████████████| 2210/2210 [00:09<00:00, 243.64it/s]

Once it is indexed you can use text queries find and curate interesting subsets. A general method to prepare these subsets is to use lunr.text.fetch. This will fetch a subset of vectors that are close in vector space and save the associated examples on disk. From there you can use any Prodigy recipe you like.

prodigy
lunr.text.fetch
examples.jsonl
index.gz.json
subset.jsonl
--query "outrage better service unhappy"

More interfaces

As a convenience this plugin also provides the textcat.lunr.manual, ner.lunr.manual and spans.lunr.manual so that you may query and annotate directly. These recipes have the same arguments as their native Prodigy textcat.manual, ner.manual and spans.manual counterparts but add a --query parameter so that you may pass your query.

Interactive Queries

Sometimes you may want to update the stream while you’re annotating. You can do that without restarting the server by using the --allow-reset flag when you’re starting the textcat.lunr.manual, ner.lunr.manual or spans.lunr.manual recipes.

prodigy
textcat.lunr.manual
examples.jsonl
index.gz.json
--query "outrage better service unhappy"
--allow-reset

API

prodigy
lunr.text.index
source
examples.index

prodigy
lunr.text.fetch
source
index_path
out_path
--query

Sense2vec

sense2vec (Trask et. al, 2015) is a nice twist on word2vec that lets you learn more interesting and detailed word vectors. This library is a simple Python implementation for loading, querying and training sense2vec models. To explore the semantic similarities across all Reddit comments of 2015 and 2019, see the interactive demo. There are also more details in this blogpost.

To see a demo on how to use this tool with Prodigy, you may enjoy this Youtube video where we use it to detect video games in text.

To use sense2vec, you’ll first need to install it.

python -m pip install sense2vec

To use the pre-trained vectors in Prodigy you’ll need to download the archive(s) and extract them. Large files have been split into multi-part downloads. All the available versions can be found below.

To merge the multi-part archives, you can run the following:

cat s2v_reddit_2019_lg.tar.gz.* > s2v_reddit_2019_lg.tar.gz

Once downloaded (and merged) you should be able to unarchive via:

tar -xvf s2v_reddit_lg.tar.gz

Now that the archive is extracted you can point the sense2vec.teach recipe to it. This will allow Prodigy to suggest similar terms based on the most similar phrases from sense2vec, and the suggestions will be adjusted as you annotate and accept similar phrases. For each seed term, the best matching sense according to the sense2vec vectors will be used.

prodigy
sense2vec.teach
video_game_yesno
/path/to/s2v_reddit_2019_lg
--seeds "mass effect,knights of the old republic,halo 3"
--resume

After curating the generated examples you can choose to export the collected phrases as pattern files which can be used with spaCy’s EntityRuler or recipes like ner.manual by using the sense2vec.to-patterns recipe.

prodigy
sense2vec.to-patterns
video_game_yesno
blank:en
VIDEO_GAME
patterns.jsonl

This will generate a patterns.jsonl file locally that has contents that may look like:

{"label": "VIDEO_GAME", "pattern": [{"LOWER": "mass"}, {"LOWER": "effect"}]}
{"label": "VIDEO_GAME", "pattern": [{"LOWER": "knights"}, {"LOWER": "of"}, {"LOWER": "the"}, {"LOWER": "old"}, {"LOWER": "republic"}]}
{"label": "VIDEO_GAME", "pattern": [{"LOWER": "halo"}, {"LOWER": "3"}]}
{"label": "VIDEO_GAME", "pattern": [{"LOWER": "jade"}, {"LOWER": "empire"}]}

More recipes

Sense2vec also has the sense2vec.eval, sense2vec.eval-most-similar and sense2vec.eval-ab recipes. These may be interesting if you’re interested in evaluating a sense2vec model. For more information on those, you can check the README on the Github repository.

prodigy
sense2vec.teach
dataset
vectors_path
--seeds
--threshold
--n-similar
--batch-size
--case-sensitive
--resume

prodigy
sense2vec.to-patterns
dataset
spacy_model
label
--output-file
--case-sensitive
--dry

🔎 Prodigy-evaluate

This Prodigy plug-in allows you to evaluate your spaCy pipeline overall or on a per-example basis. To use these recipes, you’ll first need to install the plugin.

Install prodigy-evaluatepip install "prodigy-evaluate @ git+https://github.com/explosion/prodigy-evaluate"

Once installed, you can make use of the two main recipes in this plugin: evaluate.evaluate and evaluate.evaluate-example.

Example evaluate.evaluate output

prodigy
evaluate.evaluate
my_custom_ner_model
--ner ner_dataset
--label-stats
ℹ Using CPU

================================== Results ==================================


TOK     100.00
NER P   92.80 
NER R   99.58 
NER F   96.07 
SPEED   26868


=============================== NER (per type) ===============================


                 P        R       F
SKILL        92.53    99.55   95.91
EXPERIENCE   96.88   100.00   98.41

Example evaluate.evaluate-example output

prodigy
evaluate.evaluate-example
my_custom_ner_model
--ner ner_dataset
--metric ents_f
--n-results 3
ℹ Using CPU

============================== Scored Examples ==============================


Example                     ents_f
-----------------           ------
I live in london.           0.0
My name is Freya.           0.0
Where is Antonia?           0.0

prodigy
evaluate.nervaluate
my_custom_ner_model
--ner ner_dataset
--per-label