Computing Marks from Multiple Assessors Using Adaptive Averaging

Alex Ignjatovic, Chung Tong Lee, Paul Compton, Cat Kutay, Hui Guo

Research output: Chapter in Book/Report/Conference proceedingConference Paper published in Proceedingspeer-review

Abstract

Consider a situation in which a group of assessors mark a collection of submissions; each assessor marks more than one submission and each submission is marked by more than one assessor. Typical scenarios include reviewing conference submissions and peer marking in a class. The problem is how to optimally assign a final mark to each submission. The mark assignment must be robust in the following sense. A small group of assessors might collude
and give marks which significantly deviate from the marks given by other assessors. Another small group of assessors might give arbitrary marks, uncorrelated with the others’ assessments. Some assessors might be excessively
generous while some might be extremely stringent. In each of these cases, the impact of the marks by assessors from such groups has to be appropriately discounted. Based on the work in [2], we propose a method which produces
marks meeting the above requirements. The final mark assigned to each submission is a weighted average of marks by individual assessors; the weight given to each assessor’s mark is inversely related to the total variance of all his
marks from the final marks. Clearly, such definition is circular, and the existence of a final mark assignment having such a property is proved using the Brouwer Fixed Point Theorem for continuous maps on convex compact sets [1].
We provide a fast converging iterative algorithm for computing such a fixed point and give results of empirical tests of the robustness and adequacy of the marks calculated by our algorithm.
Original languageEnglish
Title of host publicationProceedings of International Conference International Conference on Engineering Education & Research (ICEE & ICEER 2009) Korea
Number of pages7
Publication statusPublished - 2009
Externally publishedYes

Fingerprint

Dive into the research topics of 'Computing Marks from Multiple Assessors Using Adaptive Averaging'. Together they form a unique fingerprint.

Cite this