Iranian Journal of Applied Linguistics

en Construct Validation of a Rating Scale through a Training Program: A Multifaceted Rasch Analysis in Speaking Assessment Construct Validation of a Rating Scale through a Training Program: A Multifaceted Rasch Analysis in Speaking Assessment عمومى General پژوهشي Research <div dir="ltr" style="text-align: justify;">Performance testing including the use of rating scales has become highly widespread in the evaluation of second/foreign oral assessment. However, few studies have used a pre-, post-training design investigating the impact of a training program on the reduction of raters’ biases to the rating scale categories resulting in increase in their consistency measures. Besides, no study has used MFRM including the facets of test takers’ ability, raters’ severity, task difficulty, group expertise, scale category, and test version all in a single study. 20 EFL teachers rated the oral performances produced by 200 test takers before and after a training program using an analytic rating scale including fluency, grammar, vocabulary, intelligibility, cohesion and comprehension categories. The outcome of the study indicated that MFRM can be used to investigate raters’ scoring behavior and can result in enhancement in rater training and validating the functionality of the rating scale descriptors. Training can also result in higher levels of interrater consistency and reduced levels of severity/leniency; however, it cannot turn raters into duplicates of one another, but can make them more self-consistent. Training helped raters use the descriptors of the rating scale more efficiently of its various band descriptors resulting in reduced halo effect. Finally, the raters improved consistency and reduced rater-scale category biases after the training program. The remaining differences regarding bias measures could probably be attributed to the result of different ways of interpreting the scoring rubrics which is due to raters’ confusion in the accurate application of the scale. </div> <div style="text-align: justify;">Performance testing including the use of rating scales has become highly widespread in the evaluation of second/foreign oral assessment. However, few studies have used a pre-, post-training design investigating the impact of a training program on the reduction of raters’ biases to the rating scale categories resulting in increase in their consistency measures. Besides, no study has used MFRM including the facets of test takers’ ability, raters’ severity, task difficulty, group expertise, scale category, and test version all in a single study. 20 EFL teachers rated the oral performances produced by 200 test takers before and after a training program using an analytic rating scale including fluency, grammar, vocabulary, intelligibility, cohesion and comprehension categories. The outcome of the study indicated that MFRM can be used to investigate raters’ scoring behavior and can result in enhancement in rater training and validating the functionality of the rating scale descriptors. Training can also result in higher levels of interrater consistency and reduced levels of severity/leniency; however, it cannot turn raters into duplicates of one another, but can make them more self-consistent. Training helped raters use the descriptors of the rating scale more efficiently of its various band descriptors resulting in reduced halo effect. Finally, the raters improved consistency and reduced rater-scale category biases after the training program. The remaining differences regarding bias measures could probably be attributed to the result of different ways of interpreting the scoring rubrics which is due to raters’ confusion in the accurate application of the scale.</div> Bias, Interrater consistency, Intrarater consistency, Multifaceted Rasch Measurement (MFRM), Rater training, Rating scale Bias, Interrater consistency, Intrarater consistency, Multifaceted Rasch Measurement (MFRM), Rater training, Rating scale 48 80 http://ijal.khu.ac.ir/browse.php?a_code=A-10-798-8&slc_lang=en&sid=1 Wander Lowie Wander Lowie 10031947532846002725 10031947532846002725 No Department of Applied Linguistics Department of Applied Linguistics Houman Bijani Houman Bijani houman.bijani@gmail.com 10031947532846002726 10031947532846002726 Yes Department of English, Zanjan Branch, Islamic Azad University, Zanjan, Iran Department of English, Zanjan Branch, Islamic Azad University, Zanjan, Iran Mohammad Reza Oroji Mohammad Reza Oroji mohammadrezaoroji@yahoo.com 10031947532846002727 10031947532846002727 No Department of English, Zanjan Branch, Islamic Azad University, Zanjan, Iran Department of English, Zanjan Branch, Islamic Azad University, Zanjan, Iran Zeinab Khalafi Zeinab Khalafi khalafi9630@gmail.com 10031947532846002728 10031947532846002728 No Department of English, Zanjan Branch, Islamic Azad University, Zanjan, Iran Department of English, Zanjan Branch, Islamic Azad University, Zanjan, Iran Pouya Abbasi Pouya Abbasi 10031947532846002729 10031947532846002729 No Chabahar Maritime University