While Fraunhofer IVI scientists are developing an algorithm for data fusion and processing of radar data, Fraunhofer IAIS experts are working on the development of the corresponding AI model for recognition and localization (detection) of traffic participants on infrared images. They are also supporting the Fraunhofer IVI in the development of the underlying algorithm. The Fraunhofer institutes FHR and IIS not only support the implementation of the project and advise on data campaigns, but are also responsible for the proof of concept of the AI model for the classification of traffic participants.
In contrast to conventional forms of AI models for the detection of people and vehicles, the Fraunhofer approach is based on the combination of two different modalities for data acquisition: the fusion of radar and infrared data creates a robust analysis system for which even temporary occlusions of traffic participants do not cause a problem - the data acquisition is thus very reliable even in difficult lighting conditions and can thus be used regardless of the daytime, in real time and in almost any weather. Object detection is also supplemented by a tracking method that provides additional robustness and enables the analysis of traffic participants in image sequences and 3D. The collection and analysis of sensitive data from public spaces is carried out in compliance with data protection regulations.
Traffic is recorded using various sensors. Classical clustering methods are used for detection in the radar data. The evaluation of the infrared data is based on a deep learning approach with a corresponding Convolutional Neural Network (CNN). The fusion of both detection sources is realized, among others, by using a Hungarian algorithm. The choice of sensors is not standardized, but in combination is very robust to a wide range of problems. Furthermore, the sensors also complement each other in localization, so that a 3D position of traffic participants can be determined.
Both sensors are not suitable for identifying individuals, which reduces the risk of data privacy issues in the data collection phase. In addition, neither images nor videos are recorded to ensure that the privacy of traffic participants is protected.
Neither the infrared nor the radar sensor unit is directly dependent on the sun or other illumination.
Simplified, most objects in the infrared range emit in some proportion to their temperature. This self-emissivity makes objects, as well as people and (moving) motorized vehicles, visible regardless of illumination. In addition, some types of fog are less opaque in the infrared range than in the visible range, so you can see through them.
In contrast, radar is an active sensing system. This means that it emits its own signal and detects the response from the environment - regardless of the prevailing light conditions. In addition, unlike the commonly used visual system, the radar system is robust to impairments such as temporary occlusions of traffic participants.
Tailored to specific use cases, recorded traffic data can be evaluated. This allows the prediction of traffic flows or real-time communication with autonomous vehicles or driver assistance systems. Furthermore, real-time communication can be an important tool in traffic management - such as for traffic light switching - or can be used to identify free routes for emergency services.