AI marking reliability is about whether automated or AI-assisted scoring can support a real assessment decision without weakening validity, consistency, fairness, transparency, or the right to challenge a mark. The central issue is not whether a system can produce a score, but whether that score is dependable in the specific qualification context where it is used. The strongest evidence points towards cautious, hybrid use with human oversight rather than opaque full automation in high-stakes settings. Reliability has to be demonstrated for the actual task, cohort, and decision, not inferred from general model performance.

AI marking reliability is the extent to which automated or AI-assisted scoring can support an assessment decision without undermining the quality of that decision. In assessment terms, the question is whether the score is consistent, valid for its intended use, explainable to the people who need to defend it, and robust enough for moderation and appeal. In high-stakes contexts, reliability cannot be assumed from general AI performance; it has to be shown in the real assessment setting.

Scoring sits at the centre of trust in assessment. AI can reduce workload and speed up marking, but it can also make it harder to explain why marks were awarded, which affects moderation, appeals, bias review, and public confidence. That matters most where decisions have real consequences for learners, providers, and regulators.

- **Reliability**: whether similar responses are treated consistently enough for the decision being made. - **Validity**: whether the score supports the intended interpretation of learner performance. - **Explainability**: whether markers, moderators, and appeals panels can understand why a mark was produced. - **Human-in-the-loop review**: human oversight that can confirm, adjust, or override AI-supported scoring. - **Hybrid marking**: a model where AI supports workflow, feedback, or first-pass scoring, but humans remain accountable for judgement.

The source set converges on a fairly clear practical view: AI marking can be useful, but it must be tested in context and governed tightly. Public-sector guidance and implementation evidence carry more weight than supplier marketing here. Harno’s approach in Estonia is a strong example of treating AI grading as something to be evaluated inside a real exam setting, with human graders still central and both technical and substantive testing required before use. The e-Assessment Association’s essay-marking discussion points in the same direction: efficiency, consistency, and transparency only count if the human oversight and quality assurance are explicit. The new E-Assessment Association guidance on AI grading solutions also aligns with this, stressing fit to educational goals, learning outcomes, and system integration rather than speed alone. There is also broad agreement that hybrid models are more defensible than fully opaque automation. The more complex or high-stakes the decision, the more important it is that the system can be explained, moderated, escalated, and appealed. TCN guidance aligns with that direction by favouring expert-in-the-loop review and by treating explainability as more important than clever prompting. The newer comparative evidence on ChatGPT grading reinforces the same caution: general-purpose large language models may appear consistent, but they can also score slightly higher than human markers and struggle with course-specific queries. That does not rule AI marking out, but it does show why reliability and standard-setting need to be checked on the actual exam rather than assumed from fluency or consistency alone. The manual-to-AI-assisted transition resource adds a practical implementation layer: when institutions move from manual grading to AI support, they need evaluation, staff training, and continuous monitoring rather than a single go-live decision.

What remains unsettled is how far AI can go beyond support functions without losing trust. Supplier material often frames the issue in terms of efficiency, scale, feedback quality, or headline accuracy metrics, but those claims do not by themselves settle the reliability question. The open question is whether the system holds up under the exact assessment conditions that matter to the buyer. A related tension is between automation and interpretability. The field appears to be moving towards more AI support in marking workflows, but there is no clear consensus that the score itself should be handed to an opaque model in high-stakes use. The deeper assessment issue is whether the organisation can defend the mark, not just generate it. The BMC Medical Education review is directionally supportive of AI in educational outcomes, but it does not by itself resolve marking reliability. It is useful as a reminder that positive learning impacts and dependable scoring are different questions.

- **Validity risk**: a score may look consistent while measuring something slightly different from the intended construct. - **Bias risk**: edge cases, unusual responses, or subgroup effects may be handled unevenly. - **Transparency risk**: if the rationale for marks is unclear, appeals and moderation become harder. - **Governance risk**: human review routes, escalation, and audit may not match the scoring model in practice. - **Procurement risk**: headline accuracy claims may be treated as evidence when they are only vendor claims. - **Public trust risk**: learners and regulators may not accept scores they cannot interrogate or challenge.

1. Define what the learner must do unaided, and what kind of evidence the assessment is meant to produce. 2. Test the system in the specific subject, task type, and cohort where it will be used. 3. Choose the reliability measure that matches the stakes of the decision. 4. Keep humans central for review, exception handling, and override. 5. Require evidence for explainability, moderation, and appeal readiness. 6. Ask for subgroup performance, edge-case handling, and independent validation rather than only vendor-reported accuracy. 7. Separate marking support from final scoring where the evidence is not yet strong enough for direct use. 8. When introducing AI-assisted assessment workflows, build in evaluation, training, and continuous monitoring rather than a one-off implementation step.

| Option | What it means | Main strength | Main trade-off | |---|---|---|---| | Prohibit AI marking | Human markers do all scoring | Maximum interpretability and control | Higher workload and slower turnaround | | Permit AI for support only | AI helps with workflow, feedback, or first-pass triage | Easier to govern and explain | Limited efficiency gains | | Integrate AI into final scoring | AI contributes directly to marks with human oversight | Potentially larger efficiency gains | Harder to defend validity, bias, and appeals | The evidence currently supports the first two options more comfortably than the third for high-stakes use. Where direct AI scoring is considered, the burden of proof is much higher.

A certification body wants to reduce marking workload on short-answer responses. A cautious approach is to use AI for first-pass sorting and marker support, then keep final judgement with trained assessors and moderation routes. That preserves efficiency gains without making the award dependent on an opaque score that the organisation may struggle to explain later. An additional practical example is the E-Assessment Association's newly published consideration set for institutions exploring AI grading. Its emphasis on educational goals, learning outcomes, LMS integration, and institutional fit is a reminder that reliability starts with use-case definition, not with model performance alone. The manual-to-AI-assisted transition resource adds that institutions should not stop at the pilot decision: they need staff training and monitoring to keep the workflow trustworthy over time.

- Harno / ERR report on AI grading in Estonia, which shows a public-sector testing posture for e-exams. - TCN guidance on marking with AI, which emphasises explainability and expert-in-the-loop review. - Questionmark’s interview on AI-powered certification scoring, which is useful practitioner commentary on complex and high-stakes use. - e-Assessment Association essay-marking discussion, which reinforces the need for human oversight and quality assurance. - New E-Assessment Association guidance on AI grading solutions for institutions. - New E-Assessment Association resource on moving from manual to AI-assisted assessments. - Vendor pages from Top Marks AI, Stylus, SmartMarker, EduTest Pro, Crowdmark, ATI, Smartail, and Learnosity, which are useful as market signals but not as independent validation. - BERA / British Educational Research Journal comparative study of ChatGPT and human grading of higher-education exams, which supports a cautious reading of AI marking reliability. - BMC Medical Education systematic review on AI and educational outcomes in health professions education, which is useful context but not direct marking validation.

The market is crowded with tools promising automated essay marking, handwritten script handling, rubric-based feedback, workflow integration, and audit support. This is a useful signal that demand is real, but the evidential weight is uneven: vendor pages describe capability, while the stronger question is whether a particular product has been independently validated in the intended assessment setting. Learnosity’s reported QWK figure, for example, is a performance claim that needs context and independent benchmarking before it can be treated as general proof.

### What is AI marking reliability in assessment? It is the degree to which AI-supported scoring can be trusted to produce fair, consistent, and defensible marks in a specific assessment context. ### Can AI be used for marking safely? Possibly, but only where the use case, evidence, and governance are strong enough. The safest pattern in the current evidence set is hybrid use with human oversight, especially for high-stakes decisions. ### What should buyers ask suppliers? Ask for the validation method, the assessment context, reliability evidence, handling of edge cases, moderation route, appeal support, and any independent testing. A headline accuracy figure is not enough on its own. ### Why does explainability matter so much? Because marks must often be defended to learners, moderators, regulators, and appeal panels. If the rationale cannot be explained, trust in the score drops even if the tool is efficient.

Tim Burnett (Admin)

Test Community Network. "AI marking reliability." TCN AI & Assessment Wiki. Last reviewed 2026-05-03. https://www.testcommunity.network/wiki/ai-marking-reliability.html

- Top Marks AI vendor page. - Stylus vendor page. - SmartMarker vendor page. - EduTest Pro vendor page. - No More Marking post on AI-enhanced marking at scale. - ERR report on Harno and AI grading of e-exams. - TCN guidance on AI for marking. - Crowdmark vendor source. - ATI Custom Assessment Builder vendor source. - Smartail DeepGrade vendor source. - Learnosity AI-assisted scoring and feedback page. - BMC Medical Education systematic review on AI and educational outcomes in health professions education. - Questionmark interview on AI-powered certification scoring. - Principal Barker article on ChatGPT and handwritten exam marking. - e-Assessment Association AI SIG discussion on eMarking of essay questions. - BERA / British Educational Research Journal comparative study of ChatGPT and human grading in higher education. - E-Assessment Association guidance on AI grading solutions. - E-Assessment Association manual-to-AI-assisted assessment resource.