|
|
|
 |
Search published articles |
 |
|
Showing 5 results for Bijani
Houman Bijani, Mona Khabiri,
Volume 20, Issue 1 (4-2017)
Abstract
| Although the use of verbal protocols is growing in oral assessment, research on the use of raters’ verbal protocols is rather rare. Moreover, those few studies did not use a mixed-methods design. Therefore, this study investigated the possible impacts of rater training on novice and experienced raters’ application of a specified set of standards in rating. To meet this objective, the study made use of verbal protocols produced by 20 raters who scored 300 test takers’ oral performances and analyzed the data both qualitatively and quantitatively. The outcomes demonstrated that through applying the training program, the raters were able to concentrate more on linguistic, discourse, and phonological features; therefore, the extent of their agreement increased specifically among the inexperienced raters. The analysis of verbal protocols also revealed that training how to apply a well-defined rating scale can foster its use for raters both validly and reliably. Various groups of raters approach the task of rating in different ways, which cannot be explored through pure statistical analysis. Thus, think-aloud verbal protocols can shed light on the vague sides of the issue and add to the validity of oral language assessment. Moreover, since the results of this study showed that inexperienced raters can produce protocols of higher quality and quantity in the use of macro and micro strategies to evaluate test takers’ performances, there is no evidence based on which decision makers should exclude inexperienced raters solely because of their lack of adequate experience. |
Mohammad Reza Oroji , Houman Bijani, Saeid Moharrami Gheydari , Rouhollah Rajabi , Mahsa Sheikhi,
Volume 26, Issue 1 (3-2023)
Abstract
This study was an attempt to investigate the relationship between test anxiety and writing performance among Iranian intermediate EFL students. To achieve this goal, 50 participants of EFL students (male and female) at Iran, Tehran, were selected on the basis of sampling convenience. The instruments used in this research consisted of the proficiency test and TAQ (Test Anxiety) questionnaire. After homogenizing the students via a proficiency test (Cambridge Placement Test, 2010), those students who were selected as upper-intermediate ones were allowed to take part in this research. Their age range was between 20 and 40. Having collected the results, the researchers recorded the scores in computer files for statistical analysis using the Statistical Package for the Social Sciences (SPSS) version 25.00. In order to measure the relationship between the variables (test anxiety & written performance) Pearson Correlation tests were run. The results illustrated that there were significant relationships between writing performance and text anxiety and all its sub-scales of worry and emotionality. In addition, it was proved that there was not any significant difference between male and female in respect to writing performance. It must be reiterated that a Pearson Correlation test was also run to measure the Inter-Rater Reliability. The results of this study will provide insights to help EFL learners reduce their test anxiety and increase the level of their confidence.
Wander Lowie, Houman Bijani, Mohammad Reza Oroji, Zeinab Khalafi, Pouya Abbasi,
Volume 26, Issue 2 (9-2023)
Abstract
Performance testing including the use of rating scales has become highly widespread in the evaluation of second/foreign oral assessment. However, few studies have used a pre-, post-training design investigating the impact of a training program on the reduction of raters’ biases to the rating scale categories resulting in increase in their consistency measures. Besides, no study has used MFRM including the facets of test takers’ ability, raters’ severity, task difficulty, group expertise, scale category, and test version all in a single study. 20 EFL teachers rated the oral performances produced by 200 test takers before and after a training program using an analytic rating scale including fluency, grammar, vocabulary, intelligibility, cohesion and comprehension categories. The outcome of the study indicated that MFRM can be used to investigate raters’ scoring behavior and can result in enhancement in rater training and validating the functionality of the rating scale descriptors. Training can also result in higher levels of interrater consistency and reduced levels of severity/leniency; however, it cannot turn raters into duplicates of one another, but can make them more self-consistent. Training helped raters use the descriptors of the rating scale more efficiently of its various band descriptors resulting in reduced halo effect. Finally, the raters improved consistency and reduced rater-scale category biases after the training program. The remaining differences regarding bias measures could probably be attributed to the result of different ways of interpreting the scoring rubrics which is due to raters’ confusion in the accurate application of the scale.
Mohammad Reza Oroji, Houman Bijani, Saeid Moharrami Gheydari, Mahsa Sheikhi, Rouhollah Rajabi,
Volume 27, Issue 1 (4-2024)
Abstract
This study aimed to examine the speech of autistic children aged 6 to 10 in the city of Tehran, Iran based on the three metafunctions of Halliday's Systemic Functional Grammar (SFG) theory and to strengthen the connection between psychology and linguistics. The research method in this study was a combination of quantitative and qualitative approaches. Qualitatively, this study was a descriptive-analytical investigation. The statistical population of the study included all children aged 6 to 10 with autism in Tehran in 2018, from which 7 children were randomly selected using stratified sampling. Additionally, a control group of 7 non-autistic children aged 6 to 10 was randomly chosen. Since autistic individuals are generally less verbal, the researcher, with the help of a teacher, recorded and transcribed the speech of autistic children discussing specific visual topics using a mobile phone. The sentences were analyzed based on experiential, interpersonal, and textual metafunctions. Quantitatively, the differences between the two groups in terms of the use of processes, modal verbs, and negative polarity were examined using a two-tailed t-test. Descriptive results showed that the material process accounted for the highest percentage of all processes in the speech of autistic individuals, whereas the existential process accounted for approximately zero percent. This indicates that autistic individuals have difficulty understanding the concept of the existential process. Material processes focus on the occurrence of events, as they are based on our experience of the physical world. In the comparison between the two groups, material processes had the highest mean in both groups, followed by relational and mental processes.
Zahra Orouji, Houman Bijani, Mohammadreza Oroji,
Volume 28, Issue 1 (4-2025)
Abstract
As oral language proficiency assessment relies on human judgment, raters play a crucial role in performance-based testing. Among rater-related variables, rating experience has received considerable attention. Previous research on rater training has shown that extremely severe or lenient raters often benefit most from training, leading to changes in rating behavior. However, many of these studies have applied FACETS to only one or two facets and have rarely employed pre- and post-training designs. In addition, empirical findings have been inconsistent, providing no clear evidence as to whether experienced or inexperienced raters demonstrate greater rating reliability. The present study investigated the impact of rater training on experienced and inexperienced raters. Twenty raters evaluated the oral performances of 200 test takers before and after participating in a training program. The results indicated that training increased interrater consistency and reduced bias in the use of rating scale categories. The findings further suggested that, given the difficulty of fully eliminating rater variability, rater training should prioritize improving intrarater reliability rather than focusing exclusively on agreement among raters. Both experienced and inexperienced raters showed improved rating quality following training; however, inexperienced raters demonstrated greater gains. These results suggest that inexperienced raters should not be excluded from rating solely due to limited experience. As inexperienced raters are also more cost-effective, the findings imply that testing authorities may benefit more from investing in effective rater-training programs than from allocating substantial resources to recruiting highly experienced raters.
|
|
Publisher
