Abstract
Consider a situation in which a group of assessors mark a collection of submissions; each assessor marks more than one submission and each submission is marked by more than one assessor. Typical scenarios include reviewing conference submissions and peer marking in a class. The problem is how to optimally assign a final mark to each submission. The mark assignment must be robust in the following sense. A small group of assessors might collude
and give marks which significantly deviate from the marks given by other assessors. Another small group of assessors might give arbitrary marks, uncorrelated with the others’ assessments. Some assessors might be excessively
generous while some might be extremely stringent. In each of these cases, the impact of the marks by assessors from such groups has to be appropriately discounted. Based on the work in [2], we propose a method which produces
marks meeting the above requirements. The final mark assigned to each submission is a weighted average of marks by individual assessors; the weight given to each assessor’s mark is inversely related to the total variance of all his
marks from the final marks. Clearly, such definition is circular, and the existence of a final mark assignment having such a property is proved using the Brouwer Fixed Point Theorem for continuous maps on convex compact sets [1].
We provide a fast converging iterative algorithm for computing such a fixed point and give results of empirical tests of the robustness and adequacy of the marks calculated by our algorithm.
and give marks which significantly deviate from the marks given by other assessors. Another small group of assessors might give arbitrary marks, uncorrelated with the others’ assessments. Some assessors might be excessively
generous while some might be extremely stringent. In each of these cases, the impact of the marks by assessors from such groups has to be appropriately discounted. Based on the work in [2], we propose a method which produces
marks meeting the above requirements. The final mark assigned to each submission is a weighted average of marks by individual assessors; the weight given to each assessor’s mark is inversely related to the total variance of all his
marks from the final marks. Clearly, such definition is circular, and the existence of a final mark assignment having such a property is proved using the Brouwer Fixed Point Theorem for continuous maps on convex compact sets [1].
We provide a fast converging iterative algorithm for computing such a fixed point and give results of empirical tests of the robustness and adequacy of the marks calculated by our algorithm.
| Original language | English |
|---|---|
| Title of host publication | Proceedings of International Conference International Conference on Engineering Education & Research (ICEE & ICEER 2009) Korea |
| Number of pages | 7 |
| Publication status | Published - 2009 |
| Externally published | Yes |