Data annotation guidelines template

A data annotation guideline is the reference for what each label means, when to apply it, and how to handle uncertainty. For outsourced labeling, it also gives your team a shared basis for reviewing a vendor’s work.

A practical starting structure

What to include in annotation guidelines

Use this outline to write your own instructions. The paid kit provides an editable .docx template with a worked sample.

1. Project summary. Describe the task, the input data, the unit to label, the intended use, and the output format. Say what is outside the project’s scope.
2. Label taxonomy. Define every label with inclusion and exclusion rules. Explain whether labels are mutually exclusive or multiple labels can apply.
3. Correct and incorrect examples. Pair each example with a reason. Include confusing cases, rather than only obvious ones.
4. Edge cases and escalation. State what to do with ambiguous, unreadable, missing, or unsupported data. Identify who resolves questions.
5. Version log. Record the rule change, date, owner, and affected labels. Tell the vendor which version applies to each batch.

Test the rules before scaling annotation

Have a small group label the same calibration set independently. Compare disagreements against the written guideline, then revise unclear definitions and examples. Include difficult cases and record the decisions so the same question does not need to be settled twice.

Keep calibration separate from routine batch review: agreeing on the rules and estimating delivered quality answer different questions. Use the annotation QA sample size calculator when planning a simple random review sample.

Connect the guideline to your vendor workflow

Reference the guideline version in your data annotation RFP. After delivery, group review errors by type and severity, then use recurring disagreements to improve the next version. Check quality alongside annotation throughput so speed does not become the only target.

Questions about this resource