Addressing pilot respond in airborne collision avoidance algorithm based on deep reinforcement learning

Аuthors

Neretin E. S.^1*, Zuocheng L. ²

1. Integration center branch of the Irkut Corporation, 5, Aviazionny pereulok, Moscow, 125167, Russia
2. Northwestern Polytechnical University, 710072, 127, West Youyi Road, Beilin District, Xi'an Shaanxi, P.R.China

*e-mail: evgeny.neretin@ic.yakovlev.ru

Abstract

Current airborne collision avoidance systems use pseudocode or numerical tables to represent optimal strategies, achieving a high level of flight safety. However, a series of issues during the development of these systems have limited the integration of collision avoidance systems with avionics and further development in the future. These issues include inaccuracies caused by interpolation techniques, neglect of pilot response variability, and the generation of too large numerical tables used to store optimal strategies. In response to these challenges, we employ Deep Reinforcement Learning (DRL) methods to solve the collision avoidance problem and pursue a more principled approach that uses a probabilistic model to account for pilot response variability in close proximity. In this paper, we first introduced the current research status of airborne collision avoidance systems and the relevant theories of deep reinforcement learning. We then constructed a simulation environment tailored to aircraft collision avoidance problems, developed a reward system that varies with the Time to Conflict (TTC), and established a probabilistic model to handle pilot response variability. We applied the DQN (Deep Q-Network) algorithm to train an agent capable of addressing aircraft collision avoidance problems while also considering altitude coordination of the two aircrafts. Finally, we tested the algorithm's effectiveness and robustness through simulation experiments in collision avoidance scenarios of varying difficulty.

References

1. Holland J E, Kochenderfer M J, Olson W A. Optimizing the next generation collision avoidance system for safe, suitable, and acceptable operational performance[J]. Air Traffic Control Quarterly, 2013, 21(3): 275-297. 
2. De, D., and Sahu, P., “A Survey on Current and NextGeneration Aircraft Collision Avoidance System,” International Journal of Systems, Control and Communications, Vol. 9, No. 4, 2018, pp. 306–337.
3. Kochenderfer, M. J., Holland, J. E., and Chryssanthacopoulos, J. P.,“Next-Generation Airborne CollisionAvoidance System,” Lincoln Laboratory Journal, Vol. 19, No. 1, 2012, pp. 17–33.
4. Kochenderfer M J, Chryssanthacopoulos J P. Robust airborne collision avoidance through dynamic programming[J]. Massachusetts Institute of Technology, Lincoln Laboratory, Project Report ATC-371, 2011, 130.
5. Kochenderfer M J, Amato C, Chowdhary G, et al. Optimized airborne collision avoidance[J]. 2015.
6. Julian K D, Kochenderfer M J, Owen M P. Deep neural network compression for aircraft collision avoidance systems[J]. Journal of Guidance, Control, and Dynamics, 2019, 42(3): 598-608.
7. Engel Y, Mannor S, Meir R. Reinforcement learning with Gaussian processes[C]//Proceedings of the 22nd international conference on Machine learning. 2005: 201-208.
8. Munos R, Moore A. Variable resolution discretization in optimal control[J]. Machine learning, 2002, 49: 291-323.
9. Julian K D, Sharma S, Jeannin J B, et al. Verifying aircraft collision avoidance neural networks through linear approximations of safe regions[J]. arXiv preprint arXiv:1903.00762, 2019.
10. Akintunde M, Lomuscio A, Maganti L, et al. Reachability analysis for neural agent-environment systems[C]//Sixteenth international conference on principles of knowledge representation and reasoning. 2018.
11. Ivanov R, Weimer J, Alur R, et al. Verisig: verifying safety properties of hybrid systems with neural network controllers[C]//Proceedings of the 22nd ACM International Conference on Hybrid Systems: Computation and Control. 2019: 169-178.
12. Kochenderfer M J, Espindle L P, Kuchar J K, et al. A comprehensive aircraft encounter model of the national airspace system[J]. Lincoln Laboratory Journal, 2008, 17(2): 41-53.
13. J. K. Kuchar and A. C. Drumm, “The Traffic Alert and Collision Avoidance System,” Lincoln Laboratory Journal, vol. 16, no. 2, pp. 277–296, 2007.
14. Collision Avoidance System Optimization with Probabilistic Pilot Response Models
15. Bertram J, Wei P, Zambreno J. A fast Markov decision process-based algorithm for collision avoidance in urban air mobility[J]. IEEE transactions on intelligent transportation systems, 2022, 23(9): 15420-15433.
16. Mnih V, Kavukcuoglu K, Silver D, et al. Playing atari with deep reinforcement learning[J]. arXiv preprint arXiv:1312.5602, 2013.
17. Mnih V, Kavukcuoglu K, Silver D, et al. Human-level control through deep reinforcement learning[J]. nature, 2015, 518(7540): 529-533.
18. Zhao Y, Liu P, Zhao W, et al. Twice sampling method in deep Q-network[J]. Acta Automatica Sinica, 2019, 14: 1870-1882.
19. Lin L J. Reinforcement learning for robots using neural networks[M]. Carnegie Mellon University, 1992.
20. Lim Y, Gardi A, Sabatini R, et al. Avionics human-machine interfaces and interactions for manned and unmanned aircraft[J]. Progress in Aerospace Sciences, 2018, 102: 1-46.

Download