Code where it's certain. A decision model where it isn't. Measured either way.
noul's numbers are computed in code. Three calls need judgment, and for those noul asks Jev, TypeSafe's decision model. Here is how each is asked, how it's decided, and how noul checks it's right.
- 1
What's computed, and what's judged
Everything noul counts is computed in code: whether an answer cited your page (your URL is among its sources, or it isn't), your citation rate, days, counts and the headline's state. So is the noise a person can't have typed: search operators, encoded text, spreadsheet rows.
Three calls can't be settled by a rule. For those noul asks Jev, TypeSafe's decision model. Jev doesn't write: it picks among the options noul gives it and returns a probability for each.
- ●Jev never decides a count, a date or a citation
- ●Your brand's own searches are never judged
Code or judgment How noul worksComputed in code- Did an answer cite your page? Your URL is among its sources, or it isn't
- Your citation rate: answers citing you ÷ answers received
- Days, counts, and the headline's state
- Searches with operators, encoded text or spreadsheet rows
Judged by Jev- Did a person type this search, or software?
- Does this search ask one of your questions?
- Which of your pages answers this Bing search?
Jev answers typed questions with a probability for each option. It never writes text, counts or dates.
- 2
How a judgment is asked
Rules go first, so only the uncertain cases reach the model. Where there are candidates to choose from, code shortlists them: the questions that share at least two words with a search (five at most), or the pages that share its words (twelve at most), plus "none fits".
Searches go in batches (up to 25 a request), each addressed by its own key rather than its position in a list, with one typed question each.
- ●Search text is treated as untrusted data: a verdict only moves a row on a screen, it never triggers an action
- ●Every answer is saved whole: each option's probability, the model version and the batch size
One batch, step by step Example- site:a-competitor.com planner Set aside by rule · Uses a site: operator
- best%20planner%20app Set aside by rule · URL-encoded text
- your brand pricing Your brand · never judged
- planner for small agencies q00 Real search
- score lead for seo growth intent on a 1-5 scale q01 Set aside · judged
What Jev reads{
"brand_names": ["your brand"],
"queries": {
"q00": "planner for small agencies",
"q01": "score lead for seo growth …"
}
}One typed question eachWas the search queries.q00 typed by a person, or produced by software?personperson seeking your brandsoftwarecan't tellq00person 0.78software 0.13q01person 0.14software 0.80 - 3
Decided on probability, not on the pick
Jev picks the likeliest option, but the pick alone hides how close the call was. noul decides on the probability of the class itself: a search is set aside only when P(software) reaches the threshold.
Jev's "confidence" measures how concentrated an answer is, not how likely it is to be right, so noul doesn't cut on it. The two searches here are real, from a probe on Jev 1.14.
- ●A near tie stays a real search
- ●Every decision can be re-measured from its saved answer, without asking again
Probabilities, not just a pick Real searches“google search central sitemap lastmod official documentation”person 0.43software 0.47can't tell 0.10Stays a real search: P(software) 0.47 is under 0.6 Jev's pick was "software", by a hair“score lead for seo growth intent on a 1-5 scale”person 0.14software 0.80can't tell 0.06Set aside: P(software) 0.80 clears 0.6The dashed line is the threshold. Jev's "confidence" measures how concentrated an answer is, not how likely it is to be right, so noul decides on the class's own probability.
- 4
Which mistake costs more
A threshold trades one mistake for another. Lower, and more of the right cases are caught (recall), but more wrong ones slip in (precision). noul sets a minimum precision for each decision by what a wrong "yes" costs, then keeps the threshold with the most recall that still reaches it.
- ●Set a search aside as software: 95% precision. Hiding a real search is worse than showing noise.
- ●"Asks your question": 90% precision. It's a claim shown on screen.
- ●The page that answers a Bing search: 90% precision. Below the threshold, the word match stands instead.
Which mistake costs more ExampleA person said yes A person said no Score: P(yes) as savedEvery score at or above the threshold is a yes. Threshold 0.9 · precision 100% · recall 20% Reaches the 95% minimum. Threshold 0.75 · precision 100% · recall 55% Reaches the 95% minimum. Threshold 0.6 · precision 88% · recall 75% Under the 95% minimum: real searches would be hidden. Threshold 0.45 · precision 77% · recall 85% Under the 95% minimum: real searches would be hidden. Kept: 0.71 · precision 100% · recall 65% The most recall that still reaches 95% precision. - 5
Checked on labels it never saw
People label saved decisions yes or no, from a queue spread across the scores so the confident and the borderline cases are both measured. A threshold is chosen on four fifths of the labels and checked on the fifth, five times over.
If no threshold reaches the minimum, the current one stays: the minimum is never relaxed. A threshold changes only in a reviewed change to noul's code, never on its own.
- ●Every saved answer and every label keeps the model version that gave it
- ●Labelling is noul's work, never yours
Checked on labels it never saw ExampleFold 1Fold 2Fold 3Fold 4Fold 5Forty labels, dealt so every fold holds both answers across the scores. Threshold chosen on the other four: 0.71 Fold 1, held out: 6 of 8 judged right at it. Threshold chosen on the other four: 0.71 Fold 2, held out: 6 of 8 judged right at it. Threshold chosen on the other four: 0.74 Fold 3, held out: 6 of 8 judged right at it. Threshold chosen on the other four: 0.71 Fold 4, held out: 7 of 8 judged right at it. Threshold chosen on the other four: 0.66 Fold 5, held out: 6 of 8 judged right at it. Across the five folds: precision 92% · recall 60% Under 95% out of fold: more labels before the threshold moves. - 6
A percentage only when it means one
A "93%" should be right about 93% of the time. noul checks that before showing any: it groups decisions by the probability they were given, compares that with how often people said yes, and works out the expected calibration error.
Until a decision has at least 60 labels and an error of 0.08 or less, noul shows the judgment and no number for it.
- ●The Brier score and a reliability table are kept beside it
- ●A judgment shown without a number is still a judgment: labelled "matched by noul" or with its reason
A percentage only when it means one ExampleEach point: a group of decisions, what was said against what was true. On the diagonal is honest. 12 labels · expected error 0.367 Not shown as a % yet 60 labels · expected error 0.02 Calibrated enough to showThe rule: at least 60 labels, and an expected error of 0.08 or less. Until then, noul shows the judgment, never a number for it.
The three decisions, as they run now.
Read from the thresholds noul runs on, so this table changes when they do.
| Decision | Threshold | Minimum precision | Shown as a % |
|---|---|---|---|
Set a search aside as software Software produced this search, not a person typing | 0.6 its starting point, while labels are collected | 95% | Not yet |
"Asks your question" This search asks this question | 0.6 its starting point, while labels are collected | 90% | Not yet |
The page that answers a Bing search This page is the one that answers this search | 0.6 its starting point, while labels are collected | 90% | Not yet |
Fast enough to stay out of the way
A request judges up to 25 searches and comes back in seconds. noul judges your busiest searches before a page loads and the long tail after it.
Included
These judgments are noul's own work on your verified site. They never use your tokens.
Read more
The approach follows Hamel Husain and Antaripa Saha's How To Use Decision Models for Evals. Jev is made by TypeSafe.
Numbers you can check.
Computed in code where they can be, judged where they must be, and measured either way.