Effects of time lag between speech and text on understanding stories

Purpose: This study examined the psychological effects of hearing and understanding speech and the text presented after the speech. Theoretical background: Speech recognition systems are an accessible method for students with hearing loss and have been used to understand speech messages. However, previous studies have reported that speech mistranslation and text time lags by the system make it difficult for students with hearing loss to understand. However, it is unclear how time lags and text errors affect individuals with hearing loss. Research methods: Five participants with normal hearing (mean age: 26.6y) completed a speech and text comprehension task. Short stories were presented aurally by the speakers, and the text captions of the stories were presented on a computer screen at three time lag conditions: 0 (presented simultaneously), 1, and 2 seconds. The stories included 10 word errors for each condition. We asked participants to raise their hands when they found errors between the auditory presented speech and the visually presented texts. After the tasks, the participants were administered a confirmation test for the contents of the stories; their listening efforts to hear and understand the stories were assessed using a psychological scale and the results for each condition were compared. Results: The mean correct response of task was 92% (SD 7.5), 90% (8.9), and 80%（19.0） for the 0-, 1-, and 2-second conditions, respectively. The longer the time lag, the more the decrease in correct responses. Furthermore, as the time lag increased, the mean correct responses on the confirmation test decreased and the scale scores of the listening effort increased. Discussion: The results indicated that the time lag and errors between auditory and visual information affected stress and individuals’ ability to hear and understand stories. Difficulties in understanding speech messages owing to time lags and errors would increase for individuals with hearing loss. Clarifying these effects will lead to the development of better speech recognition systems for individuals with hearing loss.