Every category is checked,
twice, by humans.
The models learn from expert human annotation, then every category clears two checks: an expert-coded gold standard, and a second, blind coding. Each category then carries its own reliability tier, including the ones we report as fragile.
Out of 100 sentences, the models classify 87 the way the human expert does
The overall macro F1 is 0.866. It averages precision (are its calls right?) and recall (does it find every real case?) equally across every measured category, so rare categories count for as much as frequent ones.
Above 0.80, a category supports an analysis across the whole corpus; between 0.60 and 0.80, it gives a lead that still needs confirming; below 0.60, it supports nothing more than a commented example. The corpus's sixty-odd categories are defined in the What we extract chapter.
What is an F1 score?
The F1 score combines two things: how often the model's calls are right (precision) and how many real cases it finds (recall). It runs from 0 (worst) to 1 (perfect).
A concrete exampleA model detects the "Protest" category. Out of 100 sentences, 10 really are about protests. The model flags 12, gets 8 of the 10 real ones, and 4 of its flags are wrong:
Above 0.80, a category is considered reliably classified.
One dot per measured category
F1 score, from 0 to 1. Hover or tap a dot to read its detail.
Figure 1 — how closely the models match the human expert, category by category. Weighted F1 per category, from 0 to 1, measured on the gold standard held out of training. The gauge carries the validation's overall macro F1 (0.866); the scatter places every measured category at its own F1, and the tooltip gives its precision, its recall, its number of test sentences and the gap between English and French. Reading it: the further right a dot sits, the closer the agreement; dots below 0.60 mark the categories we report as fragile.
One thousand gold-standard sentences are coded twice, the second time blind
Our models never grade their own homework. They learn from expert human annotation, then every category is validated twice: against an expert-coded gold standard, then against a second annotator who never saw the first coding.
What this validation does not guarantee
Categories that stay below 0.60 F1 support no corpus-wide analysis: no time series, no comparison between outlets, no counts. We serve them anyway, labelled fragile, so that no one mistakes them for solid measurements.
The gold standard measures agreement on one thousand hand-coded sentences, not on the 287,227 articles in the corpus. A model trained on the vocabulary of past years can drift on new words, and the published F1 would not show it: it is computed once, on a frozen test set.