Document Type

Conference Proceeding


Australian Council for Computers in Education


School of Education




Newhouse, C., & Tarricone, P. (2016). Online moderation of external assessment using pairwise judgements. In Australian Council for Computers in Education 2016 Conference Refereed Proceedings. Brisbane, Australia.

Available here


When an assessment involves judgement by more than one assessor, it is usual to consider that some form of moderation is required to ensure consistent results. This often involves face-to-face meetings to compare judgements, and agree upon a score or grade. As well as inconsistency between assessors there is typically substantial error due to inconsistency in judgement between student performances by individual assessors. This is particularly the case where judgements are necessarily highly subjective. The traditional forms of moderation tend to be logistically difficult and analytical scoring using devices such as rubrics make generating reliable scores difficult. It is likely that problems such as these can be tackled using modern technologies, a thesis that we set out to investigate in the final phase of a three-year project into summative assessment in senior secondary schooling. The study investigated the use of digital technologies to support a form of social online moderation, which involved the use of digital communications and assessment tools to facilitate a pairwise comparative judgement approach. It involved a small group of Visual Arts teachers from rural schools in Western Australia assessing digitised forms of artworks submitted for high-stakes summative assessment at the end of Year 12. The aim was to determine whether the use of these technologies would provide good moderation outcomes and valuable professional learning for those involved. The participants were guided through the processes that involved no face-to-face meetings and were questioned about their experiences of these processes and technologies, and their attitudes and perceptions of them. In addition, the scores from their judgements were analysed. The results demonstrated that it was feasible to use these technologies to support moderation processes under these circumstances. Further participants perceived that the social online moderation processes supported by the technologies had assisted them in making more consistent judgements and increasing their understanding of the standards of work submitted and the application of the assessment criteria. They found the comparative judgement approach easy to use and appropriate for assessing the highly subjective artworks. However, the study found that probably due to the inexperience of some of the assessors the reliability of the final scores was not as high as anticipated. As a result, a slightly more rigorous method for social online moderation is recommended by the study.

Creative Commons License

Creative Commons Attribution 3.0 License
This work is licensed under a Creative Commons Attribution 3.0 License.

Included in

Education Commons