Is “It Usually Gets It Right” Enough?

01

One card with a flag

Jiwoo is sorting 1,000 cards for a community gathering. Cards with a small star pattern on the back will be collected separately. In this fictional box, exactly 10 cards have a star and 990 do not. A complete checking sheet records the actual pattern on every card.

A trial record for a sorting device says, “900 of the 1,000 cards were classified correctly.” A correct classification means saying star when a star is present and no star when it is absent. Seeing 90%, Jiwoo thinks a card flagged by the device should be fairly trustworthy.

The flagged cards pile up on a tray. What Jiwoo wants to know has now changed. Instead of asking, “How often was it right across all the cards?” she is asking, “How many cards on this tray actually have a star?” Both questions concern the same device. But do they count the same group of cards?

02

Count from two different starting groups

The fuller trial record says the device flagged 9 of the 10 cards that really had stars. It left the other one unflagged. Among the 990 cards without stars, it wrongly flagged 99 and left 891 unflagged. These four counts are fictional trial results chosen for this explanation.

The correct classifications are the 9 stars found and the 891 cards correctly classified as having no star. That makes 9+891=900, so overall accuracy really is 900÷1,000=90%. But the tray holds both the 9 genuine star cards and the 99 wrongly flagged cards. Its total is 9+99=108.

In this record, the share of flagged cards that actually have stars is therefore 9÷108=1/12, or about 8.3%. Neither 90% nor about 8.3% is an arithmetic error. The first calculation uses all 1,000 cards; the second uses only the 108 flagged cards. The groups represented by the denominators were different from the start.

03

How common was the target before the flag?

The proportion of a target in the starting population is called its base rate. In this box, the base rate of star-patterned cards is 10÷1,000=1%. Because stars were rare, wrong flags from the much larger no-star group could still contribute many cards to the tray.

“The share of actual star cards that were flagged” is 9÷10=90%. “The share of flagged cards that actually have stars” is 9÷108. Similar words appear in both sentences, but their starting conditions are reversed. Looking for a flag when a star is known to be present and looking for a star when a flag is known to be present are different questions.

This does not mean the base rate alone explains everything about a flag. We also need to know how many genuine stars the device finds and how often it flags a card without a star. Here, we counted the two starting groups and their respective flagging results, then reconstructed the new group formed by all the flags.

04

Before trusting or discarding the device

We cannot immediately carry the 1% star rate over to another box. A box of cards left after an event and a box prepared by deliberately mixing in star cards might have different compositions. Even with the same flagging method, it matters which population the trial used.

The result also changes if there are fewer wrong flags. If far fewer than 99 cards are flagged among the 990 without stars, the tray will have a different composition. Instead of looking at one number called “90% accurate,” we have a reason to separate targets missed from non-targets included by mistake.

The intended use matters as well. Gathering candidates for someone to check again is different from putting cards into a final box on the strength of a flag alone. In this record, checking only the tray would allow someone to find the 9 flagged star cards, while missing the one star left among the unflagged cards.

These counts describe the result of this fictional trial. They do not guarantee the result for the next card. Changes in the condition of the cards or the camera setup could change how the device works. Whether real records are sufficient, and whether they match the cards being sorted now, are further questions to examine.

05

When you encounter another kind of flag

A similar question can be useful when a library catalog marks books as relevant to a topic you want. The composition of the marked list depends on how common those books are in the catalog and whether unrelated books receive the label too. We cannot transfer the cards’ roughly 8.3% to the books.

Writing down “Out of which group?” can make a percentage easier to interpret. Is it the share correct across everything, the share of actual targets found, or the share of flags that are correct? What sounded like one statement that a system “usually gets it right” separates into several different descriptions.

Without the necessary records, an exact calculation may not be possible. Instead of assigning a probability without a basis, we can say that we want to know the starting proportion of targets and the record of wrong flags. Identifying the missing counts can help us decide how far our present confidence should extend.

Jiwoo looks at the note beside the tray. She brings her pencil toward the words “star cards,” then pauses. What has collected here is a set of cards the device flagged, rather than a set already confirmed to have stars. She lifts the first card and turns it over. The checking sheet and the remaining unsorted cards lie alongside her.