Veldkamp, Bernard P. ; Matteucci, Mariagiulia
(2009)
Dealing with multiple criteria in test assembly.
[Preprint]
Full text available as:
Abstract
It is quite common that tests or exams are being used for more then one purpose. First of all, they are used to measure the ability of the students in a reliable manner. Besides, they can be used for pass/fail decisions or to predict future behavior of the candidate, like future job behavior or academic performance. The question remains how to assemble a test that can be used for all these different purposes, that is, how to assemble a multi-objective test. Besides, multiple objectives can result from different purposes, but also from the way test specifications have been implemented. For the WDM-model, for multidimensional IRT, for Cognitive Diagnostic CAT, but also for infeasibility analysis, multiple objective test assembly problems have to be solved.
In this paper, a 2-stage method is presented for dealing with multiple objectives in test assembly. In the normalization stage, all objectives are brought on a common scale. In the valorization stage, the different objectives are being compared and related to each other. The method is applied to a Guidance Test developed at the University of Bologna, and a comparison is made with more traditional single objective test assembly methods. The results clearly demonstrate the importance and relevance of multi-objective test assembly.
Abstract
It is quite common that tests or exams are being used for more then one purpose. First of all, they are used to measure the ability of the students in a reliable manner. Besides, they can be used for pass/fail decisions or to predict future behavior of the candidate, like future job behavior or academic performance. The question remains how to assemble a test that can be used for all these different purposes, that is, how to assemble a multi-objective test. Besides, multiple objectives can result from different purposes, but also from the way test specifications have been implemented. For the WDM-model, for multidimensional IRT, for Cognitive Diagnostic CAT, but also for infeasibility analysis, multiple objective test assembly problems have to be solved.
In this paper, a 2-stage method is presented for dealing with multiple objectives in test assembly. In the normalization stage, all objectives are brought on a common scale. In the valorization stage, the different objectives are being compared and related to each other. The method is applied to a Guidance Test developed at the University of Bologna, and a comparison is made with more traditional single objective test assembly methods. The results clearly demonstrate the importance and relevance of multi-objective test assembly.
Document type
Preprint
Creators
Keywords
automated test assembly, multiple criteria, multiple objectives, simulated annealing, guidance test
Subjects
DOI
Deposit date
04 Nov 2009 09:49
Last modified
16 May 2011 12:11
URI
Other metadata
Document type
Preprint
Creators
Keywords
automated test assembly, multiple criteria, multiple objectives, simulated annealing, guidance test
Subjects
DOI
Deposit date
04 Nov 2009 09:49
Last modified
16 May 2011 12:11
URI
Downloads
Downloads
Staff only: