Next generation interfaces for pilots: integration of eye-tracking, gesture, and voice control systems

Аuthors

Pisarev I. A., Radchenko P. S., Iosifov P. A.^*, P.

Moscow Aviation Institute (National Research University), 4, Volokolamskoe shosse, Moscow, А-80, GSP-3, 125993, Russia

*e-mail: iosifovpa@mai.ru

Abstract

The study addresses contemporary methodologies for the design of multimodal piloting interfaces that integrate gaze-tracking technologies, gesture-recognition systems, and voice-based command inputs. The analysis demonstrates that the synergistic fusion of these modalities can substantially enhance operator effectiveness while mitigating cognitive load. A structural model describing human–machine interaction within next-generation interfaces is introduced, and the potential applicability of these technologies in adaptive ergatic control systems for aircraft is evaluated.

Keywords:

multimodal interfaces; piloting; gaze tracking; voice control; gesture recognition; ergonomics; cognitive load

References

1. Palmov S.V. Kashapova D.R. Aviatsionno-kosmicheskaya tekhnika i tekhnologii, 2025, no. 4. pp. 209–216.

2. Galishnikov Yu.A.. Nedelya nauki SPbPU: materialy nauchnoi konferentsii s mezhdunarodnym uchastiem, 18–23 noyabrya 2019 g. Inzhenerno-stroitel'nyi institute [SPbPU science week: Proceedings of the International Scientific Conference, November 18–23, 2019. Institute of Civil Engineering]. Saint Petersburg, Politekh-press, 2020, pt. 3, pp. 256–258. Avialable at: https://elibrary.ru/item.asp?id=42514050.

3. Dvornikov M.V., Bogomolov A.V., Kuchevsky S.V. Izvesti TulGu. Tekhnicheskie nauki, 2025, no. 2, pp. 458–465. Avialable at: https://cyberleninka.ru/article/n/metod-avtomatizirovannogo-kontrolya-sostoyaniya-letchika-v-polete

4. Wickens C.D. Multiple resources and performance prediction. Theoretical Issues in Ergonomics Science, 2002, vol. 3, no. 2. pp. 159–177. DOI 10.1080/14639220210123806.

5. Endsley M.R. Toward a theory of situation awareness in dynamic systems. Human Factors, 1995, vol. 37, no. 1. pp. 32–64. DOI 10.1518/001872095779049543.

6. Parasuraman R., Riley V. Humans and automation: Use, misuse, disuse, abuse. Human Factors, 1997. vol. 39, no. 2. pp. 230–253. DOI 10.1518/001872097778543886.

7. Rieth M., et al. Adaptable automation for a more human-centered work environment. Safety Science, 2024, vol. 186, art. 103246. Avialable at: https://www.sciencedirect.com/science/article/pii/S1071581924000302.

8. Fu B., et al. Event-Driven Predictive Gaze Analytics to Enhance Pilot Performance // ACM Proceedings. 2025.

9. Yang Zhang, Jie Chen, Xiao Zhong, Yongcong Shao, Weiwei Fu. Evaluating the effectiveness of flight simulator training on perceptual-motor skills. Scientific Reports, 2025, vol. 5, iss. 1, art. 28062. DOI 10.1038/s41598-025-12929-0.

10. Alreshidi I., Moulitsas I., Jenkins K.W. Multimodal approach for pilot mental state detection based on EEG. Sensors, 2023, vol. 23, art. 7350. DOI 10.3390/s23177350.

11. Quynh Anh Nguyen, ORCID,Nam Anh Dao, Long Nguyen. Enhanced pilot attention monitoring: A time-frequency EEG analysis using CNN–LSTM Networks for aviation safety. Information, 2024, vol. 16 (6), art. 503. DOI 10.3390/info16060503.