|
|
|
|
|
 |
Search published articles |
 |
|
Showing 3 results for Performance Assessment
Gholam Reza Kiany, Monireh Norouzi,
Volume 19, Issue 2 (9-2016)
Abstract
Performance assessment is exceedingly considered a key concept in teacher education programs worldwide. Accordingly, in Iran, a national assessment system was proposed by Farhangian University to assess the professional competencies of its ELT graduates. The concerns regarding the validity and authenticity of traditional measures of teachers' competencies have motivated us to devise a localized performance assessment scheme. Therefore, the present study aimed to develop a performance assessment scheme to be used as a benchmark for assessing the professional competencies of ELT graduates of this university. To this end, three assessment tasks and rating scales were developed, piloted, and administered. Next, Haertel's participatory approach was employed to set passing standards for the assessment tasks as well as the whole assessment scheme. Analysis of the data revealed inter-rater and intra-rater reliability coefficients of 0.85 and 0.89. The validity of the assessment scheme was also confirmed by experts' judgments made, to a large extent, on the correspondence between the target domain and test domain skills. Based on the results, the proposed assessment scheme is rendered more efficient and reliable in comparison to traditional tests with regard to the following dimensions: a) higher degrees of reliability and validity of the assessment scheme aimed at the improvement of licensure and program development; b) stronger evidence for inter-/intra- rater reliability and consistency of scoring; and c) an optimized and systematic procedure for setting passing standards based on the consensus of experts' judgments. It is believed that further development of the proposed assessment scheme unlocks its potential to be used as a large-scale teacher assessment model for Farhangian University.
Houman Bijani, Mona Khabiri,
Volume 20, Issue 1 (4-2017)
Abstract
| Although the use of verbal protocols is growing in oral assessment, research on the use of raters’ verbal protocols is rather rare. Moreover, those few studies did not use a mixed-methods design. Therefore, this study investigated the possible impacts of rater training on novice and experienced raters’ application of a specified set of standards in rating. To meet this objective, the study made use of verbal protocols produced by 20 raters who scored 300 test takers’ oral performances and analyzed the data both qualitatively and quantitatively. The outcomes demonstrated that through applying the training program, the raters were able to concentrate more on linguistic, discourse, and phonological features; therefore, the extent of their agreement increased specifically among the inexperienced raters. The analysis of verbal protocols also revealed that training how to apply a well-defined rating scale can foster its use for raters both validly and reliably. Various groups of raters approach the task of rating in different ways, which cannot be explored through pure statistical analysis. Thus, think-aloud verbal protocols can shed light on the vague sides of the issue and add to the validity of oral language assessment. Moreover, since the results of this study showed that inexperienced raters can produce protocols of higher quality and quantity in the use of macro and micro strategies to evaluate test takers’ performances, there is no evidence based on which decision makers should exclude inexperienced raters solely because of their lack of adequate experience. |
Zahra Mohammadi Salari,
Volume 27, Issue 1 (4-2024)
Abstract
The current study explored the status of rating scales among Iranian EFL raters. It appeared that EFL/ESL assessment environments were significantly influenced by the perceived authority of native assessment groups. Consequently, examining the realities of rating practices in EFL/ESL settings could offer a more accurate understanding of how assessment is viewed and implemented. To assess this, the present study conducted a comprehensive survey within the Iranian EFL writing assessment framework. A carefully designed eight-item interview guide was created to investigate various aspects of the rating task, including the rating scale. This guide was administered to ten raters from various universities and institutions in Iran, all of whom possessed either a Master's or Doctorate degree in TEFL. The raters participated in 40 minutes interview sessions, and the audio-recorded interviews were transcribed by the researcher for qualitative analysis. Through a thorough content analysis of the interview data, some general patterns emerged. Results of interviews with Iranian EFL composition raters revealed that rating scale in its common sense did not exist. In fact, raters relied on their own internalized criteria developed through their long years of practice. Therefore, native speaker legitimacy in the design and development of scales for the EFL context was challenged and the local agency in the design and development of rating scale was emphasized.
|
|
|
|
|
|
|
|
|
Publisher
