Multispectral panoramic video images forming technology for aviation optical-electronic vision systems

Аuthors

Kudinov I. A.^*, Kholopov I. K.^**, Khramov M. Y.^*

Ryazan State Instrument-making Enterprise, 32, Seminarskaya str., Ryazan, 390000, Russia

*e-mail: hunter@grpz.ryazan.ru
**e-mail: kholopov.i.s@rsreu.ru

Abstract

The article considers the video image forming technology according to the information from distributed multi-spectral cameras of the aviation panoramic optical-electronic vision system. It analyses the main problems of a panoramic frame stitching quality reduction while working with multi-spectral cameras. The article presents the geometric formulation of the problem and main analytical expressions describing the spherical panorama forming procedure without evaluating the point features of the scene, and searching matches between them using descriptors. A robust to shooting conditions algorithm for panorama image forming according to the results of preliminary photometric calibration of multispectral cameras with the special test-object and information on the angle orientation of the reference camera, obtained from the inertial micro-electromechanical sensor, was developed. One of the possible variants of the universal test-object realization for the multi-spectral cameras calibration is presented. The main operation modes of a panoramic vision system prototype with television and thermal cameras developed by the authors are considered including vision improving functions such as blending and contrasting according to Multiscale retinex algorithm, as well as information integration from the technical vision channels operating in various spectral areas. The article shows that computations parallelization using CUDA technology allows realizing vision improvement functions including information integration from multispectral sensors and overlapping additional signographic information for two independently controlled 1024 x 768 pixels regions of interest with a frequency not less than 30 Hz. The results of semi-natural experiments on the window of user interest display in the “transparent cabin” mode are presented.

Display of video information in the region of interest in accordance with the concept of “transparent cab” are presented.

Keywords:

spherical panorama, region of interest, image fusion, blending, CUDA technology

References

1. Bel’skii A., Zhosan N., Brondz D., Gorbachev K., Grebenshchikov V., Kargaev A. Fotonika, 2013, vol. 38, no. 2, pp. 80 – 86.

2. AN/AAQ-37 Distributed Aperture System (DAS) for the F-35, available at: http://www.northropgrumman.com/Capabilities/ ANAAQ37F35/Pages/default.aspx

3. IronVision^TM, available at: http://elbitsystems.com/media/IronVision.pdf

4. Knyaz’ V.V., Busurin V.I. Trudy MAI, 2015, no. 81, available at: http://trudymai.ru/eng/published.php?ID=57706

5. Kostyashkin L.N. Nikiforov M.B. Obrabotka izobrazhenii v aviatsionnykh sistemakh tekhnicheskogo zreniya (Image Processing in Aviation Vision Systems), Moscow, Fizmatlit, 2016, 234 p.

6. Bondarenko A., Bondarenko M. Sovremennaya elektronika, 2017, no. 1, pp. 32 – 37.

7. Ikeda S., Sato T., Yokoya N. A calibration method for an omnidirectional multi-camera system, Proc. SPIE Electronic Imaging, 2003, vol. 5006, pp. 499 – 507.

8. Gorbulin V.I., Khodor M.A. Trudy MAI, 2018, no. 100, available at: http://trudymai.ru/eng/published.php?ID=93426

9. Hartley R., Zisserman A. Multiple View Geometry in Computer Vision, Cambridge, Cambridge University Press, 2003, 656 p.

10. Brown M., Lowe D. Automatic Panoramic Image Stitching using Invariant Features, International Journal of Computer Vision, 2007, vol. 74(1), pp. 59 – 73.

11. Gevorkov S.Yu., Il’ichev O.A., Ivanchenko D.L. Pavlyuchenkov A.V. Patent 2601165 RU, 10.09.16.

12. Szeliski R. Image alignment and stitching: a tutorial, Foundations and trends in computer graphics and vision, 2006, vol. 2(1), pp. 1 – 104.

13. Kudinov I.A., Pavlov O.V., Kholopov I.S., Khramov M.Yu. III Mezhdunarodnaya konferentsiya i molodezhnaya shkola “Informatsionnye tekhnologii i nanotekhnologii”. Sbornik trudov (Samara, 25-27 April 2017), Samara, Izd-vo “Novaya tekhnika”, 2017, pp. 1580 – 1586.

14. Mishin A.Yu., Kiryushin E.Yu., Obukhov A.I., Gurlov D.V. Trudy MAI, 2013, no. 70, URL: http://trudymai.ru/eng/published.php?ID=44533

15. Chelnokov Yu.N. Kvaternionnye i bikvaternionnye modeli i metody mekhaniki tverdogo tela i ikh prilozheniya. Geometriya i kinematika dvizheniya (Quaternion and Biquaternion Models and Solid Mechanics Methods and their Applications. Geometry and Kinematics of Motion), Moscow, FIZMATLIT, 2006, 512 p.

16. Brown D.C. Close-Range Camera Calibration, Photogrammetric Engineering, 1971, vol. 37(8), pp. 855 – 866.

17. Burt P., Adelson E. A Multiresolution Spline with Application to Image Mosaics, ACM Transactions on Graphics, 1983, vol. 2(4), pp. 217 – 236.

18. Kudinov I.A., Pavlov O.V., Kholopov I.S., Khramov M.Yu. IV Mezhdunarodnaya konferentsiya i molodezhnaya shkola “Informatsionnye tekhnologii i nanotekhnologii”. Sbornik trudov (Samara, 24-27 Apri 2018), Samara, Izd-vo “Novaya tekhnika”, 2018, pp. 568 – 575.

19. Sanders J., Kandrot E. CUDA by Example, New York, Addison-Wesley, 2010, 290 p.

20. Zhang Z. A Flexible New Technique for Camera Calibration, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, vol. 22(11), pp. 1330 – 1334.

21. St-Laurent L., Mikhnevich M., Bubel A., Prevost D. Passive Calibration Board for Alignment of VIS-NIR, SWIR and LWIR Images, Quantitative InfraRed Thermography Journal, 2017, vol. 14(2), pp. 193 – 205.

22. Kudinov I.A., Pavlov O.V., Kholopov I.S. Patent 2672466 RU, 14.11.2018.

23. Kostyashkin L.N., Pavlov O.V. Obozrenie armii i flota, 2015, no. 2, pp. 64 – 66.

24. Kostyashkin L.N., Pavlov O.V., Tpofimov D.V. Mekhatronika, avtomatizatsiya, upravlenie, 2009, no. 3, pp. 57 – 63.

25. Jobson D.J., Rahman Z., Woodell G.A. A Multiscale Retinex for Bridging the Gap between Color Images and the Human Observation of Scenes, IEEE Trans. on Image Processing, 1997, vol. 6(7), pp. 965 – 976.

26. MATLAB Jet Array, available at: https://www.mathworks.com/help/matlab/ref/jet.html

27. Moreland K. Diverging Color Maps for Scientific Visualization (Expanded), Proc. of 5th International Symposium of Advances in Visual Computing (ISVC-2009), Part II, Las Vegas, 2009, pp. 92 – 103.

28. Li G., Xu S., Zhao X. Fast color-transfer-based image fusion method for merging infrared and visible images, Proc. of the SPIE, 2010, vol. 7710, pp. 77100.

29. Maket videopanoramy iz 5 kamer i teplovizora, available at: https://www.youtube.com/watch?v=HHdj0JcKpI8

Download