Inter-Annotator Agreement: What It Tells You About Your AI Dataset (and What It Doesn't)

If two experts label the same image and produce different results, which one is right? Sometimes neither, and sometimes both. Before a model ever sees the data, that question is worth answering, because a model trained on inconsistent labels learns the inconsistency.

Inter-annotator agreement (often shortened to IAA) is how teams measure it. It sounds technical, but the idea is simple, and anyone preparing a dataset for scientific or medical AI can use it.

What inter-annotator agreement is

Take a sample of your data and have two or more annotators label it independently, without seeing each other's work. Then compare the results. The more often they match, the higher the agreement.

This tells you something a single annotator's work never can: whether the task, as written, can be done consistently by different people. If it can't, the problem is usually the task definition rather than the people.

Common ways to measure it

The right metric depends on the type of labeling.

• Simple percent agreement. The share of items where annotators gave the same label. Easy to understand, but it can look better than it is, because some matches happen by chance.
• Cohen's kappa (two annotators) and Fleiss' kappa (three or more). These correct for agreement that would happen by chance, which makes them more trustworthy than raw percentages for classification tasks.
• Overlap measures for outlines and regions. For tasks like drawing the boundary of a structure, teams commonly compare how much two annotators' shapes overlap, using measures such as intersection over union (IoU) or the Dice coefficient.

There is no single number that counts as good enough for every project. What is acceptable depends on how difficult the task is and what the data will be used for. Treat published rules of thumb as rough guides, and agree on a target with your team before you start.

What low agreement is telling you

Low agreement is not a failure. It is a signal, and it usually points to one of a few causes:

• The guidelines are unclear. Classes overlap, or the instructions don't say what to do in borderline cases.
• The task is genuinely ambiguous. Some images simply don't have one correct answer, and that is worth knowing before you build on them.
• Annotators need more training or expertise. Specialized data often requires domain knowledge that general instructions can't replace.
• The data has quality problems. Blurry, poorly lit, or inconsistent images make everyone less certain.

What high agreement does not prove

High agreement is reassuring, but it does not guarantee the labels are correct. If every annotator follows the same flawed guideline, or shares the same misunderstanding, they will agree with each other and still be wrong. Agreement measures consistency. Accuracy needs an expert check, ideally against a trusted reference, and the two should be treated as separate questions.

A practical way to use it

You don't need a complicated setup. A simple cycle works for most projects:

1. Write the guidelines with examples and rules for hard cases.
2. Double-label a sample. Have at least two annotators label the same small portion of the data independently.
3. Measure agreement with a metric suited to your task.
4. Review the disagreements together. This is where most of the learning happens. A senior reviewer or domain expert can decide the correct answer.
5. Update the guidelines and tell the annotators what changed.
6. Repeat on a fresh sample, and keep checking a small share of the data throughout the project so quality doesn't slip.

Keep a record of the scores, the disagreements you found, and the changes you made. That record is part of what makes a dataset trustworthy, because it shows how the labels were produced.

Questions to ask your data partner

• Do you double-label a sample, and how is agreement measured?
• How are disagreements resolved, and who has the final say?
• Will I see the agreement results, not just a summary claim?
• How do guideline changes get recorded and applied to earlier work?

The takeaway

Inter-annotator agreement is a simple, low-cost check that turns "we think our labels are consistent" into something you can measure. Use it early, read low scores as a sign to improve your guidelines, and remember that consistent is not the same as correct.