Does self-paced listening increase fairness in foreign language listening tests?

Abstract Summary

With foreign language (FL) classrooms becoming increasingly heterogeneous, teachers are facing the challenge of how to best adapt assessment practices to the needs of neurodiverse learners. Providing appropriate test accommodations for neurodiverse learners is particularly challenging in the case of listening comprehension (Taylor & Khalifa, 2013). Traditional group-administered listening tests pose particular challenges for learners with reading-related learning difficulties, who have difficulties in processing phonological information (Geva & Massey-Garrison, 2013) alongside reduced working memory capacity and less efficient metacognitive skills (Kormos & Smith, 2024). Accordingly, research shows that learners with low-level literacy skills in the first language (L1) demonstrate decreased FL listening comprehension (Authors, 2023; Kormos et al., 2019) and higher levels of anxiety (Piechurska-Kuciel, 2008). Therefore, teachers are sometimes advised to allow learners with reading-related learning difficulties to self-pace, i.e., stop and rewind, the recording (e.g., BMBWF, 2021). However, this suggestion is not based on conclusive evidence: Authors (2023) found that self-paced listening did not significantly increase the L2 listening test scores of young learners with low-level L1 literacy skills and even disadvantaged learners with decreased L1 reading accuracy. Nevertheless, learners reported that they felt significantly less anxious in this listening condition (Authors, under review). This study will expand research on self-paced listening by addressing these questions:

• RQ1: To what extent is there a differential effect of self-pacing on (a) listening test scores and (b) anxiety in learners with varying L1 literacy skills, FL vocabulary knowledge, and metacognitive awareness in listening? 
• RQ2: How do learners with varying L1 literacy skills, FL vocabulary knowledge and metacognitive awareness in listening use self-pacing options in listening tests? 

A sample of 250 upper-secondary English learners in grades 11/12 took a subset of items from a standardized listening exam at B2-level in a counter-balanced design. In session 1, participants took an anchor task and two listening tasks in four different conditions regarding administration mode (administrator-controlled vs. self-paced listening) and task format (multiple-choice vs. constructed response), while their interaction with the audio player was being tracked. After each task, participants responded to selected questionnaires on test-taking strategies (based on Winke & Lim, 2014), listening anxiety (based on Elkhafaifi, 2005), self-pacing strategies, and the Metacognitive Awareness Listening Questionnaire (Vandergrift et al., 2006). In session 2, participants completed standardized L1 reading tasks targeting reading fluency, accuracy and comprehension. The Updated Vocabulary Levels Test (Webb et al., 2017) served as a proxy for FL proficiency. The differential effect of the listening conditions on test scores and anxiety was analyzed using Generalized Linear Mixed-Effects Modelling in R. The audio-tracking data was analyzed for self-pacing use (pausing, rewinding, time on task, areas visited) and listening patterns (Roussel, 2011). 

By exploring the effect of self-paced listening, this project contributes to long-overdue empirical research into test accommodations (Taylor & Banerjee, 2023). Hence, it will exemplify how research may inform classroom testing practices to increase fairness for neurodiverse learners.