AOAnnotation Ops Kit

Data annotation QA templates and tools for teams that outsource labeling

Accept the batch. Or send it back, with the math.

Check annotation quality, track delivery, and compare labeling vendors with spreadsheets and playbooks for in-house ML and data teams.

Explore the kit · from $99 Free sheet (.zip)

Sample size

The vendor's expected or past accuracy, not your target. Unsure? Use 50% for the most conservative (largest) sample.
Items to review 188 7.8% of 2,400 items
How this is calculated

n₀ = z² × p × (1 − p) ÷ e²
n = n₀ ÷ (1 + (n₀ − 1) ÷ N)
rounded up, at most N

Batch decision · paid QA Tracker

Required sample
188
Reviewed
192
Passed
183
Critical errors
0 of 0 allowed
Observed accuracy
95.3%
Wilson 95% interval
91.3% to 97.5%
85%target 95%100%
ACCEPT

Full sample reviewed, no critical errors, and 95.3% meets the 95% target.

Example data from the kit's worked sample.

Open the free annotation QA sample size calculator for the formula, a worked example, and guidance on choosing a random sample.

Three jobs, one kit

Check each batch

Size the sample, log the errors, and get an accept or reject decision you can show the vendor.

QA Core and up

Track delivery

Log output per person and cycle time per batch, so a late delivery points to a slow stage.

Ops Kit and up

Compare vendors

Pick or replace a vendor with written requirements, the same questions and the same pilot.

Ops Kit and up

Pricing

One-time purchase through Gumroad. Sales tax or VAT is added at checkout where it applies.

QA Core
$99 one-time

Check each batch.

  • Annotation Guideline Template
  • QA System playbook
  • QA Tracker & Sampling Calculator

Single-user license.

Buy QA Core
Ops Kit Full kit
$199 one-time

Check, track and compare vendors.

  • Everything in QA Core
  • Throughput Tracker (requires Google Sheets)
  • Vendor RFP Kit, 4 parts

Single-user license.

Buy Ops Kit
Ops Kit + Team
$299 one-time

One vendor process for a team.

  • Everything in Ops Kit
  • Up to 10 people at one company

Team license. No resale, redistribution or public posting.

Buy Team

Delivered as a zip of .xlsx and .docx files with a README. Terms of sale and license

The files

Real snapshots. Example data is fictional. Click a picture to see it full size.

Edge cases table from the guideline worked sample, such as a mug on a hook labeled as cup
Annotation Guideline TemplateWorked sample, edge cases. Example data.

The written bar you check the vendor's work against.

View details
  • Project summary, label definitions, correct and wrong examples, edge cases, version log
  • Not final until 3 people label the same 50 items and agree
  • Doc (.docx), QA Core and up
Results table from the QA System worked sample ending in the decision Accept, fix the 9 failed items
QA System playbookWorked sample, batch decision. Example data.

The rules you hold the vendor to, agreed before the first batch.

View details
  • Quality targets, one QA owner, severity levels, pass/fail rules
  • The loop: sample, review, score, feed errors back into the guideline
  • Worked examples for sample size, kappa and IoU
  • Doc (.docx), QA Core and up
Sampling Calculator tab: 188 items to review for a 2,400-item batch
QA Tracker & Sampling CalculatorSampling Calculator tab. Example inputs.

Checks each batch and keeps the records you need to push back.

View details
  • Sampling Calculator: items to review, accuracy interval, Accept / Reject / Incomplete decision
  • Error Log with severity and type, shared Taxonomy
  • Agreement: Cohen's kappa, Fleiss' kappa for 3 annotators, box IoU
  • Annotator Scorecard, Batch Log, Reviewer Log
  • Sheet (.xlsx), works in Excel or Google Sheets. QA Core and up
Bottleneck view showing share of cycle time per stage
Throughput TrackerBottleneck view.

Shows where the time goes, per person and per stage.

Requires Google Sheets
View details
  • Daily Log: items, hours, rework, AHT, scheduled hours, utilization
  • Per-Person Summary, Weekly Report with charts, Cycle Time, Bottleneck
  • Room for 4,997 Daily Log entries. Cycle Time is filled in for 997 batches and can be copied down to row 5,000
  • Some summary formulas are Google-only, so in Excel those cells won't update. Ops Kit and up
Weighted Scoring Sheet with knockouts and pilot gates
Vendor RFP KitWeighted Scoring Sheet.

A fair, written way to pick or switch vendors.

View details
  • Requirements Template: testable Must / Should / Nice requirements
  • Vendor Questionnaire: 30 questions every vendor answers the same way
  • Scoring Sheet: weighted scores out of 100, knockouts, price per accepted item
  • Pilot Checklist: same data and same blind test for every shortlisted vendor
  • 3 Docs + 1 Sheet, Ops Kit and up
Pilot Checklist first page
Pilot ChecklistPart of the Vendor RFP Kit.

Ends in a one-page decision note you can defend.

View details
  • Acceptance test sized with the Sampling Calculator
  • Results flow into the Scoring Sheet
  • Doc (.docx), Ops Kit and up
View methods

Standard, published methods. Every formula is visible in its cell.

Cochran sample size with finite population correction
How many random items to review for a batch size, confidence and margin.
Wilson score interval
A range for true batch accuracy that stays sensible near 100%.
Cohen's kappa, Fleiss' kappa for 3 annotators
Agreement after removing what you'd expect by chance. Landis and Koch cut-offs at 0.60 and 0.80 set the bands.
IoU for boxes
Overlap of two bounding boxes, to check annotators drew the same box.
Weighted scoring with knockouts
Criteria scored 1 to 5 and weighted to 100. A Must-have under 3 or a missed pilot gate removes a vendor before ranking.

Data annotation operations resources

Build a clearer labeling workflow

Practical guides and kit previews for writing instructions, comparing vendors, and tracking delivery.

Track annotation operations

Annotation throughput tracker

Track labeling output, rework, hours, and batch cycle time.

View throughput tracker

Choosing a provider? Read how to evaluate data annotation vendors.

About and FAQ

Annotation Ops Kit is an independent publisher.

Who is the Annotation Ops Kit for?

In-house ML and data teams managing outsourced data annotation. Use the guideline template to define labels, the QA files to review vendor batches, and the Ops Kit to track throughput and compare vendors. The kit contains spreadsheets and documents; it does not provide annotation workers or data collection services.

How many annotations should I review from a vendor batch?

It depends on the batch size, confidence level, margin of error, and expected accuracy. Use the free annotation QA sample size calculator to plan a simple random sample. Sampling estimates quality; the batch acceptance rules and critical-error limits still need to be agreed with your vendor.

What format are the files?

A zip of .xlsx and .docx files plus a README. They open in Google Sheets and Docs or in Excel and Word. The Throughput Tracker needs Google Sheets for live results.

Is there a free version?

Yes. The free sheet has the same sample size formula plus a shared error taxonomy. No email needed. Download it (.zip, 17 KB)

Are you affiliated with a labeling vendor or platform?

No. There are no affiliate or referral deals, and we don't sell labeling.

What's your refund policy?

Not useful? Within 14 days of purchase, contact us through your Gumroad receipt for a full refund, no questions asked.

Can I get an invoice?

Yes. Gumroad handles sales tax or VAT, and you can generate an invoice with your company details from your receipt.