Item Fit Analysis for Evaluating Academic Writing Performance With Rasch Measurement
DOI:
https://doi.org/10.51278/aj.v6i1.1144Keywords:
Item Fit, Academic Writing, Rasch MeasurementAbstract
This study aims to evaluate item fit through the application of the Rasch model. The research involved 40 English as a foreign language student who took part in an essay writing class as part of a TOEFL iBT course, with writing samples gathered in partnership with the language center offering the TOEFL iBT classes. The research methodology utilizes the Rasch model with Winsteps software for quantitative analysis, employing three key statistical outputs: a "statistical summary output" to provide overall data and figures, item statistics to assess item validity, and participant statistics to evaluate participant validity. Based on the interpretation of misfit order Rasch measurement for item fit, four items ("Content," "Structure," "Mechanic," and "Diction") demonstrated excellent alignment with the Rasch model. All items exhibited Outfit Mean Square (MNSQ) values below the expected 1.0 yet within the acceptable range (0.5 - 1.5), indicating better-than-expected fit, with "Diction" showing the highest conformity. The Outfit Z-Standard (ZSTD) values for all items were near zero, signifying no significant deviations from the model. High Point Measure Correlations (PT Measure Corr) of 0.91 for "Content" and "Diction" and 0.88 for "Structure" and "Mechanic" suggest strong consistency with the overall measured ability, affirming the significant and valid contributions of these items to the scale's measurement objectives. Thus, all four items are deemed excellent within the applied measurement scale, particularly "Diction," which exhibits the highest model fit.
Keywords: Item Fit, Academic Writing, Rasch Measurement
References
Meisel K. et al. 2017. “Subjectivity of Teacher Judgments: Exploring Student Characteristics That Influence Teacher Judgments of Student Ability.” Science Direct 65:48–60.
Tesio Luigi et al. 2024. “Interpreting Results from Rasch Analysis 2. Advanced Model Applications and the Data-Model Fit Assessment.” Taylor & Francis Online 46(3):604–17.
Andrich, D. 1978. “A Rating Formulation for Ordered Response Categories.” Psychometrika 43(1):561–573.
Erguvan, Inan Deniz, and Beyza Aksu Dunya. 2020. “Analyzing Rater Severity in a Freshman Composition Course Using Many Facet Rasch Measurement.” Language Testing in Asia 10(1).
Huang, Hung Yu. 2023. “Modeling Rating Order Effects Under Item Response Theory Models for Rater-Mediated Assessments.” Applied Psychological Measurement 47(4):312–27.
Jacobs., Holly. L., Stephen, A., Zingkgraf., Deanne. R., Wormuth, V., Faye, H., Jane, B., Hughey. 1981. Testing ESL Composition: A Practical Approach. Rowley: Newbury House Publishers, Inc.
Li, Guangming, Yuxi Pan, and Weijun Wang. 2021. “Using Generalizability Theory and Many-Facet Rasch Model to Evaluate In-Basket Tests for Managerial Positions.” 12(July):1–10.
Linacre, J. M. (. 2002. “What Do Infit and Outfit Mean-Square and Standardized Mean?” Rasch Measurement Transaction 16:878.
Misbach, I. H., & Sumintono, B. 2014. “Pengembangan Dan Validasi Instrumen ‘Persepsi Siswa Tehadap Karakter Moral Guru’ Di Indonesia Dengan Model Rasch.” . . PROCEEDING Seminar Nasional Psikometri 148–162.
Rahman, Yenni Arif. 2023. “Person and Item Validity and Reliability in Essay Writing Using Rasch Model.” 15(1):41–55.
Rahman, Yenni Arif, Nurhayati S., Fiza Asri Fauziah Habibah, and Fadilah Fadilah. 2024. Holistic Rubric Validity and Reliability in Essay Assessment Using Rasch Model in Blended Learning Program. Atlantis Press SARL.
Reise, S. .. 1990. “A Comparison of Item- and Person-Fit Methods of Assessing Model-Data Fit in IRT.” Applied Psychological Measurement 14(2):127–37.
Rost, J., & von Davier, M. 1994. “A Conditional Item-Fit Index for Rasch Model.” Applied Psychological Measurement 18(2):171–82.
Stemler, Steven E., and Adam Naples. 2021. “Rasch Measurement v. Item Response Theory: Knowing When to Cross the Line.” Practical Assessment, Research and Evaluation 26(May):1–16.
Sumintono, B. & Widhiarso, W. 2013. Aplikasi Model Rasch Untuk Penelitian Ilmu-Ilmu Sosial. Trim Komunikata Publishing House.
Tan, S. 2013. “Validation of an Analytic Rating Scale for Writing: A Rasch Modeling Approach.” Tabaran Institute of Higher Education. Iranian Journal of Language Testing 3(1).
Uto, Masaki. 2022. “A Bayesian Many-Facet Rasch Model with Markov Modeling for Rater Severity Drift.” Behavior Research Methods (October).
