Pin-Hua Chen

Multimodal Feedback for Enhancing Situational Awareness in Auditory-Limited Environments: A VR-Based Study on Visual and Haptic Cues

This study aims to evaluate visual and haptic warning feedback in auditory-limited environments, with a focus on its potential to support situational awareness through reaction time, accuracy, cognitive load, and trust. Through mixed-methods research with 16 participants, the results showed that Visual + Vibration feedback was associated with faster responses, lower perceived workload, and higher trust. However, it did not provide a significant additional benefit for accuracy compared with the Visual Cue condition. Overall, the findings suggest that visual and haptic feedback may play complementary roles, but multimodal feedback was not universally superior to single-modality feedback.

Background

According to the World health organization (2026), over 430 million people worldwide experience disabling hearing loss and require rehabilitation, and the numbers are expected to be more than 700 million by 2050. Road traffic crashes also cause 1.35 million deaths and over 50 million injuries or disabilities each year, with pedestrians accounting for 22% of all deaths (Road Accident Statistics | 2026 Edition, n.d.). People with hearing impairments face greater pedestrian safety risks because of limited access to auditory cues from environmental sounds and reduced situational awareness (Schwartz et al., 2022). This issue is not limited to individuals with hearing impairment, as even temporary or simulated hearing reduction has been shown to negatively affect situational awareness (Golob et al., 2024). From a UX perspective, this highlights the need to design warning systems that can communicate environmental hazards effectively without relying only on auditory information.

Aim and Research Questions

This study aims to evaluate visual and haptic warning feedback in auditory-limited environments, with a focus on its potential to support situational awareness through reaction time, accuracy, cognitive load, and trust.

RQ1: How do different feedback conditions affect reaction time and accuracy?
RQ2: How do different feedback conditions affect users’ cognitive load?
RQ3: How do different feedback conditions affect users’ trust in the warning system?

Prototype Development

This process was followed by the five-stage Design Thinking process, comprising Empathise, Define, Ideate, Prototype, and Test.

① ② Empathise & Define

Before developing the prototype, I interviewed two participants from Taiwan with hearing loss to understand their experiences as pedestrians. An interview was conducted before obtaining ethical approval; all data were excluded from the formal analysis. Based on insights from Findlater et al. (2019), I adopted a smartwatch-like form for the wearable device to support its integration into everyday use.

③ Ideation

2. VR Prototype Settings

At the beginning of each trial, the participant’s position was recorded. A motorcycle was then generated 40 m behind the participant, triggering the warning.

3. Data Collection in VR

Reaction time and accuracy were developed by operational definitions and automatically recorded by the VR system.

⑤ Test

A pilot test was conducted to refine the experimental procedure before the experiment.

Methodology

Results and Discussion

RQ1: How do different feedback conditions affect reaction time and accuracy?

Result (Reaction Time): The No Cue condition was excluded. A paired-samples t-test showed a significant difference between the Visual Cue (M = 1.37, SD = 0.30) and Visual + Vibration Cue (M = 1.19, SD = 0.25) conditions, with p = .012. This indicates that participants responded significantly faster with Visual + Vibration Cue than with Visual Cue.

Result (Accuracy): A Friedman test showed a significant difference across the three conditions, with p < .001. However, post-hoc comparisons showed no significant difference between the Visual Cue and Visual + Vibration conditions.

Interpretation: Vibration acted as an additional attention-capturing cue when visual information alone was not sufficient.

“The vibration could act as an extra cue to alert you, so you could immediately look at the radar, see where the vehicle was coming from, and then take evasive action (P9).”

“I kept holding it up during the experiment, the visual cue was quite noticeable (P5).”

Discussion: Vibration may complement visual feedback by capturing attention, rather than consistently improving performance when visual information is already sufficient.

RQ2: How do different feedback conditions affect users’ cognitive load?

Result (Cognitive Load): A Friedman test showed a significant difference across the three conditions, with p = .001. However, post-hoc comparisons showed no significant difference between the Visual Cue and No Cue conditions.

Interpretation: Visual warnings provide contextual information but require attention and interpretation, while vibration can provide immediate sensory cues to help redirect attention.

“I have to look at the radar to determine where the hazard is and confirm it, which feels less efficient when I have short reaction time (P16).”

Discussion: Combining the visual and haptic feedback may help distribute information-processing demands and reduce perceived workload.

RQ3: How do different feedback conditions influence users’ trust in the warning system?

Result (Trust): A Repeated-measures ANOVA showed a significant effect of warning condition on trust, with p < .001. Post-hoc comparisons were also conducted and showed significant differences between all three conditions.

Interpretation: Adding vibration made warnings more noticeable, potentially allowing more time to respond, while appropriate warning timing may also influence trust.

“Then I would have more time, so I would feel more confident (P5).”

Discussion: Adding vibration may have helped participants notice warnings earlier and left more time to respond, potentially resulting in higher trust.

Overall Contribution

Limitations and Further research