Define the labels
Data annotation guidelines template
Define labels, examples, edge cases, and version history.
Read the guideCompare labeling vendors
Data annotation RFP template
Ask every vendor the same questions, run a comparable pilot, and score the evidence against requirements you agreed before the proposals arrived.
A data annotation request for proposal turns your labeling task into a consistent brief for potential vendors. It should explain the work, define acceptance, and make the proposals comparable—not just collect a price per label.
Write a brief vendors can answer
What to put in a data labeling RFP
- Task and data. Describe the modality, label taxonomy, input format, output schema, languages, and expected volume. Share a representative sample you are authorized to disclose.
- Quality requirements. Attach the annotation guidelines, define error severity, and state how review, rework, and acceptance will work.
- Delivery and staffing. Ask about the proposed team, training, review ownership, capacity assumptions, milestones, and escalation route.
- Data handling. Ask where data will be accessed, whether subcontractors are involved, how access is controlled, and how deletion will be evidenced. Set requirements that fit your data and obligations.
- Commercial terms. Request the pricing unit, minimum commitments, review and rework charges, and assumptions that could change the quote.
- Pilot and decision. State the test conditions, evidence to return, decision criteria, and submission deadline.
Fictional selection example
Agree on the scorecard before ranking vendors
Separate mandatory requirements from weighted preferences. A high overall score should not override a failed requirement your team considers essential.
| Criterion | Weight | Evidence to request |
|---|---|---|
| Pilot quality | 40% | Reviewed output and error breakdown |
| Delivery plan | 25% | Staffing, capacity assumptions, milestones |
| Review and rework process | 20% | QA ownership, escalation, correction workflow |
| Commercial fit | 15% | Comparable quote with assumptions and charges |
For a 1–5 rating scale, a weighted contribution is rating ÷ 5 × weight. Add the contributions for a score out of 100. Define what each rating means and cite the evidence behind it.
These weights are examples, not a universal vendor-selection standard. Choose your own criteria and mandatory gates before evaluating bids.
Use a consistent pilot and cost basis
For the full selection workflow, read how to evaluate data annotation vendors, from the initial brief to the decision note.
Give shortlisted vendors the same representative task, guideline version, deadline, and output format. Where practical, review the returned labels without showing the reviewer which vendor produced them. Record quality, time, rework, and open questions against the same criteria.
Compare costs on the same basis. If one pilot costs $600 and 900 of its 1,000 submitted items are accepted, that is approximately $0.67 per accepted item. Compare with other vendors only when the cost inclusions and acceptance rules are equivalent.
Use the free QA sampling calculator to plan a random review sample. Define your acceptance rules separately. After selection, use the annotation throughput tracker to compare planned delivery with actual work.
The downloadable files
Put the workflow into practice
The Vendor RFP Kit includes a requirements template with Must / Should / Nice priorities, a 30-question vendor questionnaire, a weighted scoring sheet with knockouts and price per accepted item, and a pilot checklist. It is included in Ops Kit with the QA files and Throughput Tracker.
The page above is a free guide and preview. The editable kit files are a paid download through Gumroad.
Questions about this resource
What is in the downloadable RFP kit?
Three Word documents and one spreadsheet: Requirements Template, Vendor Questionnaire, Scoring Sheet, and Pilot Checklist. They are included in Ops Kit ($199) and Team ($299).
Does this page recommend an annotation vendor?
The kit is vendor-neutral and contains no vendor rankings or referrals. It helps your team document and compare evidence from the vendors you shortlist.
Can this cover data collection vendors too?
The selection structure can be adapted, but the kit focuses on data annotation. A collection brief also needs project-specific sourcing, consent and usage permissions, participant criteria, coverage, and capture requirements.
Continue the workflow
Track annotation operations
Annotation throughput tracker
Track labeling output, rework, hours, and batch cycle time.
Read the guidePlanning a QA review? Use the free sample size calculator or download the free sample-size sheet and error taxonomy.