Building a Better Mousetrap: Replacing Subjective Writing Rubrics with More Empirically-Sound Alternatives for EFL Learners
- 1 English Education, Department of Liberal Arts Education (LAEC), Ju Si-Gyeong College, Pai Chai University, Daejeon, South Korea
- 2 English Education, Department of Liberal Arts Education (LAEC), Ju Si-Gyeong College, Pai Chai University, Daejeon, South Korea
Abstract
Although writing rubrics can provide valuable feedback, the criteria they use are often subjective, which compels raters to employ their own tacit biases. The purpose of this study is to see if discreet empirical characteristics of texts can be used in lieu of the rubric to objectively assess the writing quality of EFL learners. The academic paragraphs of 38 participants were evaluated according to several empirically calculable criteria related to cohesion, content, and grammar. Values were then compared to scores obtained from holistic scoring by multiple raters using a multiple regression formula. The resulting correlation between variables (R = .873) was highly significant, suggesting that more empirical, impartial means of writing evaluation can now be used in conjunction with technology to provide student feedback and teacher training.
- American Council on the Teaching of Foreign Languages (2012). ACTFL proficiency guidelines. URL (last checked 1 October 2012). http://actflproficiencyguidelines2012.org/writing
- Beyreli, L., & Ari, G. (2009). The use of analytic rubric in the assessment of writing performance: Inter-rater concordance study. Educational Sciences: Theory & Practice, 9, 105-125.
- Brown, H. D. (2004). Language assessment principles and classroom practices. White Plains, NY: Pearson Education.
- Chodorow, M., & Burstein, J. (2004). Beyond essay length: Evaluating e-rater’s performance on TOEFL essays. Princeton, NJ: ETS.
- Cope, B., Kalantzis, M., McCarthey, S., Vojak, C., & Kline, S. (2011). Technology-mediated writing assessments: Principles and processes. Computers & Composition, 28, 79-96. doi:10.1016/j.compcom.2011.04.007
- Educational Testing Service (2008). TOEFL IBT test: Independent writing rubrics (scoring standards). URL (last checked 1 October 2012). http://www.ets.org/Media/Tests/TOEFL/pdf/Independent_Writing_Rubrics_2008.pdf
- Fang, Z., & Wang, Z. (2011). Beyond rubrics: Using functional language analysis to evaluate student writing. Australian Journal of Language and Literacy, 34, 147-165.
- Graham, S., Hebert, M., & Harris, K. R. (2011). Throw ‘em out or make ‘em better? State and district high-stakes writing assessments. Focus on Exceptional Children, 44, 1-12.
- Halliday, M. A. K., & Hasan, R. (1976). Cohesion in English. London: Longman.
- Hasan, R. (1984). Coherence and cohesive harmony. In J. Flood (Ed.), Understanding reading comprehension (pp. 181-219). Delaware: International Reading Association.
- Hoey, M. (1996). Patterns of lexis in text. New York, NY: Oxford University Press.
- Hunter, K., & Docherty, P. (2011). Reducing variation in the assessment of student writing. Assessment & Evaluation in Higher Education, 36, 109-124. doi:10.1080/02602930903215842
- Johansson, V. (2008). Lexical diversity and lexical density in speech and writing: A developmental perspective. Working Papers, 53, 61-79.
- Johnson, D., & Van Brackle, L. (2012). Linguistic discrimination in writing assessment: How raters react to African American “errors,” ESL errors, and standard English errors on a state-mandated writing exam. Assessing Writing, 17, 35-54. doi:10.1016/j.asw.2011.10.001