|
Using Multifaceted Rasch Analysis to Explore the Impact of Rater Expertise in Speaking Assessment
|
Zahra Orouji   , Houman Bijani * , Mohammadreza Oroji |
| Department of English, Zanjan Branch, Islamic Azad University, Zanjan, Iran |
|
|
Abstract: (810 Views) |
As oral language proficiency assessment relies on human judgment, raters play a crucial role in performance-based testing. Among rater-related variables, rating experience has received considerable attention. Previous research on rater training has shown that extremely severe or lenient raters often benefit most from training, leading to changes in rating behavior. However, many of these studies have applied FACETS to only one or two facets and have rarely employed pre- and post-training designs. In addition, empirical findings have been inconsistent, providing no clear evidence as to whether experienced or inexperienced raters demonstrate greater rating reliability. The present study investigated the impact of rater training on experienced and inexperienced raters. Twenty raters evaluated the oral performances of 200 test takers before and after participating in a training program. The results indicated that training increased interrater consistency and reduced bias in the use of rating scale categories. The findings further suggested that, given the difficulty of fully eliminating rater variability, rater training should prioritize improving intrarater reliability rather than focusing exclusively on agreement among raters. Both experienced and inexperienced raters showed improved rating quality following training; however, inexperienced raters demonstrated greater gains. These results suggest that inexperienced raters should not be excluded from rating solely due to limited experience. As inexperienced raters are also more cost-effective, the findings imply that testing authorities may benefit more from investing in effective rater-training programs than from allocating substantial resources to recruiting highly experienced raters.
|
|
| Keywords: Bias, Interrater consistency, Intrarater consistency, Multifaceted Rasch measurement (MFRM), Rater expertise |
|
|
|
Type of Study: Research |
Subject:
General
Received: 2026/01/27 | Accepted: 2026/02/7 | Published: 2025/04/20
|
|
|
|
|
|
|
| Send email to the article author |
|
|
|
| Add your comments about this article |
|
|
|
Publisher
