Exercising Professional Judgment in AI-Assisted Assessment: Selective Mediation of AI-Generated Evaluation by English Lecturers
DOI:
https://doi.org/10.59110/edutrend.1033Keywords:
AI-Assisted Assessment, Assessment Literacy, Generative Artificial Intelligence, professional judgment, Teacher AgencyAbstract
This qualitative case study examines how English lecturers exercise professional judgment when generative artificial intelligence (GenAI) is introduced into student writing assessment and feedback. Ten English lecturers at Universitas Khairun participated, producing 30 documented judgment episodes across three student writing samples. Each episode linked an initial lecturer assessment, AI-generated information from ChatGPT, Claude, or Gemini, and a final judgment. Data from assessment records, follow-up interviews, student texts, AI-generated outputs, and contextual documents were examined through within-episode and cross-episode thematic analysis. The findings show that AI did not move lecturers' scores in a single direction. Rather, it prompted them to retain, lower, or raise initial judgments after reconsidering the evidence. Lecturers evaluated AI output in relation to construct alignment, evidentiary weight, learner needs, task purpose, local context, and feedback usability. Professional agency was evident in how they determined which parts of the AI output could influence the final assessment. The study describes this process as selective evaluative mediation and argues that GenAI introduces a second object of professional judgment: lecturers judge not only student performance but also the relevance, usability, and authority of AI-generated interpretations of that performance.
References
Asadi, M., Ebadi, S., & Mohammadi, L. (2025). The impact of integrating ChatGPT with teachers’ feedback on EFL writing skills. Thinking Skills and Creativity, 56, 101766. https://doi.org/10.1016/j.tsc.2025.101766
Barrot, J. S. (2026). Generative artificial intelligence for automated essay scoring: Exploring teacher agency through an ecological perspective. Assessing Writing, 67, 100990. https://doi.org/10.1016/j.asw.2025.100990
Chen, X., & Pi, X. (2026). Assessing EFL students’ GenAI-assisted writing: Teachers’ pains, perceptions and practices. Assessing Writing, 67, 100996. https://doi.org/10.1016/j.asw.2025.100996
Crosthwaite, P., & Sun, S. (2026). Generative AI and L2 written feedback studies: A scoping review. RELC Journal, 57(1), 207–219. https://doi.org/10.1177/00336882251386530
Cui, Y., Meng, Y., & Tang, L. (2025). Reconsidering teacher assessment literacy in GenAI-enhanced environments: A scoping review. Teaching and Teacher Education, 165, 105163. https://doi.org/10.1016/j.tate.2025.105163
Derakhshan, A., Molana, K., & Nazari, M. (2026). “I wonder who I am when my assessment ideas are not respected”: A case-study inquiry into the role of institutional policies in English language teachers’ assessment identity. Asia Pacific Journal of Education, 46(1), 116–131. https://doi.org/10.1080/02188791.2024.2328237
Ding, L., Zou, D., & Kohnke, L. (2025). ChatGPT as an automated writing evaluation tool: How students perceive it and how it affects their writing. Education and Information Technologies, 30, 26777–26799. https://doi.org/10.1007/s10639-025-13775-3
Eedelouei, S. (2026). Factors influencing language teachers’ judgements and decision-making about the use of generative AI tools: A systematic review. Teaching and Teacher Education, 171, 105333. https://doi.org/10.1016/j.tate.2025.105333
Escalante, J., Pack, A., & Barrett, A. (2023). AI-generated feedback on writing: Insights into efficacy and ENL student preference. International Journal of Educational Technology in Higher Education, 20, 57. https://doi.org/10.1186/s41239-023-00425-2
Guo, K., & Wang, D. (2024). To resist it or to embrace it? Examining ChatGPT’s potential to support teacher feedback in EFL writing. Education and Information Technologies, 29, 8435–8463. https://doi.org/10.1007/s10639-023-12146-0
Han, J., & Li, M. (2024). Exploring ChatGPT-supported teacher feedback in the EFL context. System, 126, 103502. https://doi.org/10.1016/j.system.2024.103502
Jia, W., & Pun, J. (2025). Exploring EMI teachers’ agency in addressing language-related challenges with ChatGPT: A multiple case study. Computer Assisted Language Learning. Advance online publication. https://doi.org/10.1080/09588221.2025.2511061
Jiang, Y., Xie, L., & Cao, X. (2025). Exploring the effectiveness of institutional policies and regulations for generative AI usage in higher education. Higher Education Quarterly, 79(4), e70054. https://doi.org/10.1111/hequ.70054
Kementerian Pendidikan, Kebudayaan, Riset, dan Teknologi. (2024). Panduan penggunaan generative artificial intelligence pada pembelajaran di perguruan tinggi. Direktorat Pembelajaran dan Kemahasiswaan, Direktorat Jenderal Pendidikan Tinggi, Riset, dan Teknologi.
Lin, S., & Crosthwaite, P. (2024). The grass is not always greener: Teacher vs. GPT-assisted written corrective feedback. System, 127, 103529. https://doi.org/10.1016/j.system.2024.103529
Lu, Q., Yao, Y., Xiao, L., Yuan, M., Wang, J., & Zhu, X. (2024). Can ChatGPT effectively complement teacher assessment of undergraduate students’ academic writing? Assessment & Evaluation in Higher Education, 49(5), 616–633. https://doi.org/10.1080/02602938.2024.2301722
Mansouri, B., Molana, K., & Nazari, M. (2021). The interconnection between second language teachers’ language assessment literacy and professional agency: The mediating role of institutional policies. System, 103, 102674. https://doi.org/10.1016/j.system.2021.102674
Muñoz Muñoz, B. C., Nassaji, H., & Bello Carrillo, F. I. (2025). ChatGPT-generated versus human direct corrective feedback on L2 writing. System, 134, 103805. https://doi.org/10.1016/j.system.2025.103805
Nazari, M., De Costa, P. I., & Karimpour, S. (2026). The role of institutional policy in English language teacher autonomy, agency, and identity: A poststructural perspective. Language Teaching Research, 30(2), 885–907. https://doi.org/10.1177/13621688221143476
Phung, D. V., & Michell, M. (2022). Inside teacher assessment decision-making: From judgement Gestalts to assessment pathways. Frontiers in Education, 7, 830311. https://doi.org/10.3389/feduc.2022.830311
Pretorius, M., & Thewissen, J. (2026). Mediating L2 writing revision: The impact of teacher-moderated ChatGPT feedback on L2 accuracy and lexical diversity. Journal of Second Language Writing, 71, 101286. https://doi.org/10.1016/j.jslw.2026.101286
Steiss, J., Tate, T., Graham, S., Cruz, J., Hebert, M., Wang, J., Moon, Y., Tseng, W., Warschauer, M., & Olson, C. B. (2024). Comparing the quality of human and ChatGPT feedback of students’ writing. Learning and Instruction, 91, 101894. https://doi.org/10.1016/j.learninstruc.2024.101894
Vogt, K., Bøhn, H., & Tsagari, D. (2024). Language assessment literacy. Language Teaching, 57(3), 325–340. https://doi.org/10.1017/S0261444824000090
Wang, Y., Hu, W., & Guo, Q. (2026). GPT versus teacher feedback on English writing: An analysis of feedback characteristics and student engagement. System, 138, 103967. https://doi.org/10.1016/j.system.2025.103967
Xu, Y. (2019). English language teacher assessment literacy in practice. In X. Gao (Ed.), Second handbook of English language teaching (pp. 517–539). Springer. https://doi.org/10.1007/978-3-030-02899-2_22
Xu, Y., & Brown, G. T. L. (2016). Teacher assessment literacy in practice: A reconceptualization. Teaching and Teacher Education, 58, 149–162. https://doi.org/10.1016/j.tate.2016.05.010
Xu, Y., & Brown, G. T. L. (2017). University English teacher assessment literacy: A survey-test report from China. Papers in Language Testing and Assessment, 6(1), 133–158.
Xu, Y., & Liu, Y. (2009). Teacher assessment knowledge and practice: A narrative inquiry of a Chinese college EFL teacher’s experience. TESOL Quarterly, 43(3), 492–513. https://doi.org/10.1002/j.1545-7249.2009.tb00246.x
Xu, Y., Zhong, J., & Han, Y. (2026). How teacher feedback literacy changed by using generative artificial intelligence (GenAI) feedback as exemplars: Case studies of English as a foreign language school teachers in China. Teaching and Teacher Education, 176, 105501. https://doi.org/10.1016/j.tate.2026.105501
Yao, Y., Zhu, X., Xiao, L., & Lu, Q. (2025). Secondary school English teachers’ application of artificial intelligence-guided chatbot in the provision of feedback on student writing: An activity theory perspective. Journal of Second Language Writing, 67, 101179. https://doi.org/10.1016/j.jslw.2025.101179
Zhu, S., Li, J., Yao, Y., Guan, Y., & Zhu, X. (2025). What they provide and how: An intervention study on pre-service teachers’ GenAI-assisted writing feedback. The Internet and Higher Education, 67, 101040. https://doi.org/10.1016/j.iheduc.2025.101040
Zou, S., Guo, K., Wang, J., & Liu, Y. (2025). Investigating students’ uptake of teacher- and ChatGPT-generated feedback in EFL writing: A comparison study. Computer Assisted Language Learning. Advance online publication. https://doi.org/10.1080/09588221.2024.2447279
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Nurhadi Hamka, Akmal Jaya, Ikmal Muhammad, Nur Fajrhi, Nursastri Nurdin, Dhini Yustia Widhyah Saputri
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
the Creative Commons Attribution-ShareAlike 4.0 International License 