How many items should you check from your vendor's batch?
Enter the batch size, your confidence level, the margin of error you can accept and the accuracy you expect. You get the number of randomly chosen items to review.
Prefer a spreadsheet? Download the free sheet + error taxonomy (.zip)
Got your number? Next: does the batch pass?
This calculator tells you how many items to check. It does not tell you whether the vendor's batch is good enough. The paid kit does:
- QA Core, $99: the accuracy interval on what you reviewed (Wilson score), an Accept / Reject / Incomplete batch decision, Cohen's and Fleiss' kappa, box IoU, an error log, annotator scorecards, the guideline template and the QA System playbook.
- Ops Kit, $199: everything in QA Core, plus the throughput tracker and the vendor RFP kit to pick or switch vendors.
Cochran sample size with finite population correction
- Step 1
- n₀ = z² × p × (1 − p) ÷ e², where z comes from the confidence level (95% → 1.960), p is the expected accuracy and e is the margin of error.
- Step 2
- n = n₀ ÷ (1 + (n₀ − 1) ÷ N), where N is the batch size. This shrinks the sample for small batches.
- Step 3
- Round up, and never more than the batch itself. Review that many randomly chosen items.
Example: 2,400 items, 95% confidence, ±3%, 95% expected accuracy gives n₀ = 202.7, corrected n = 187.0, so you review 188 items. Use 50% expected accuracy when unsure. It gives the largest, most conservative sample. This is a planning number for a simple random sample. Source: NIST/SEMATECH e-Handbook of Statistical Methods.
Annotation Ops Kit is vendor-neutral and not affiliated with any labeling vendor or platform.