How to test whether AI marking is accurate enough for your papers

Step by step

1. Set papers aside before marking them. Scan real papers while they are still clean, then mark the originals as usual.
2. Choose a sample that is hard on purpose. Use the checklist below, including papers you would dread marking.
3. Record the teacher's marks per question. Use a spreadsheet with one row per student and one column per question.
4. Run the tool the way you really would. Use your normal answer key or rubric and your normal settings.
5. Label every difference. Find each question where the marks differ, and check it against the page.
6. Apply a rule you wrote in advance. Decide where the tool can be used, where a teacher reviews its marks, and where a teacher still marks by hand.

Why test an AI grader on your own papers?

Your own papers are the only evidence that fits your students, because handwriting, scan quality, subject and question style all affect how an AI grader performs. A figure measured on someone else's papers can't tell you how a tool copes with a class that writes in faint pencil, or a physics paper full of units and crossed-out working.

When a company quotes an accuracy figure, ask which papers it was measured on and who marked them for comparison. Then test anyway.

What should go in your test sample?

Your sample should be harder than an average class set, because easy papers make every tool look good. Include:

• The messiest handwriting in the class, and the faintest pencil
• Every question type you set: multiple choice, blanks, numbers, short answers and long answers
• Changed answers, two options marked on a one-answer question, and questions left empty
• Answers squeezed into a margin or continued on another page
• A strong paper, a weak paper, and work sitting on a grade boundary
• A poor phone photo, if teachers will use phones in practice

Why compare question by question instead of totals?

You should compare per question because two errors can cancel out and leave a total that looks perfect. In this made-up example, the teacher and the tool both give 14 out of 20, yet they disagree on two of the four questions.

How do you tell a reading error from a judgement error?

A reading error means the tool saw the wrong answer, and a judgement error means it saw the right answer and marked it wrongly. To tell them apart, find what the tool says the student wrote and compare that with the page. If a tool won't show you what it read, count that against it.

Which disagreements matter most?

The disagreements that matter most are confident errors nobody would notice, especially ones that change a result.

• Unflagged errors outrank flagged ones. A flagged question costs a quick look, while an unflagged error can reach the student.
• Errors that change an outcome outrank their size. One mark across a pass line matters more than two marks mid-range.
• Patterns outrank one-offs. Look for errors clustering on one question type, or leaning one way, such as always harsh on short answers.
• Errors that follow particular students are a fairness problem. If the hardest handwriting keeps losing marks, a class average won't show it.

What counts as accurate enough?

Accurate enough depends on what the marks are used for, so decide your rule before you look at the results. The table gives starting suggestions to adapt.

For written answers, measure the tool against how much teachers differ. Ask a second teacher to mark the same papers without seeing the first marks, and use the gap between them as your yardstick, since two careful teachers often differ on an essay.

How do you keep a teacher in the loop?

Keep a teacher responsible for the final marks by turning flags on for work that matters, reviewing everything flagged and checking a sample of everything else. Flags are off by default, and switching off Mark everything under Marking options turns them on.

• Read flagged questions before releasing any marks
• Spot-check a few unflagged papers, chosen at random, every time
• Give students an easy way to query a mark, since queries often reveal missed errors
• Test again after any change: a new subject, scanner, question style or version of the tool

How do you run this test in evaleasy?

To run this test in evaleasy, upload your sample as a batch with your real key and put its CSV beside your own marks, since it has one column per question. Each report shows the answer as read, the reason for the mark and how each question was marked. With a key, multiple choice and true/false are compared in code, so a wrong mark there points to the reading or the key.

Common questions

How many papers do I need to test?

There is no fixed number, and one full class set is a sensible starting point. A 40-question multiple choice paper gives you 40 comparisons per student, while an essay gives you one, so test more papers when the work is mostly written.

Can I test with papers already marked in ink?

Only if you also have a clean copy. Ticks, crosses and totals on the page can confuse a tool or hand it the answer. Use photocopies taken before marking, or run the test on your next set of papers.

Should I run the same paper through twice?

Yes, because different marks for identical work are hard to defend. In evaleasy an identical file reuses its first reading, so a repeat run tests the written marking. To test the reading again, upload a fresh scan or use Redo, which reads the paper again from scratch.

Is it all right to test with real students' papers?

Check your school's data rules first, and find out where the papers go. evaleasy sends papers to AI model providers through OpenRouter to be read and marked, and keeps files until you delete them, so delete test papers when you finish.

Can AI marking be used for official exams?

That is a decision for your exam board or school, so check its current rules first. Whatever they say, a teacher should stay responsible for marks that affect a student's future.